Collecting and preparing Textual Data for Classification


lesson1

Lesson 1: Introduction to Textual Data Collection in NLP

Blast off into the world of Textual Data Collection! 🚀 You’re about to unlock the secrets of data science in NLP. Let’s decode the mysteries together!

Introduction to NumPy Arrays

NumPy 数组简介

Let’s delve into Python’s NumPy library and focus on the centerpiece of NumPy - arrays. NumPy, an acronym for ‘Numerical Python’, specializes in efficient computations on arrays. Arrays in NumPy are more efficient than typical Python data structures.
让我们深入探讨 Python 的 NumPy 库，并将重点放在 NumPy 的核心组件—— arrays 上。NumPy 是“Numerical Python”的缩写，专门用于对数组进行高效计算。NumPy 中的数组比典型的 Python 数据结构更高效。

Meet NumPy 邂逅 NumPy

The power of NumPy lies in its fast computations on large data arrays, making it crucial in data analysis. Before we start, let’s import it:
NumPy 的强大之处在于它能够对大型数据数组进行快速计算，这使其在数据分析中至关重要。在开始之前，让我们先导入它：

Python

CopyPlay

1# Import NumPy as 'np' in Python 2import numpy as np

np is a commonly used representation for numpy.
np 是 numpy 的常用表示。

Understanding NumPy Arrays

理解 NumPy 数组

NumPy arrays, like a sorted shopping list, allow for swift computations. Arrays offer quick access to elements. Let’s create a simple one-dimensional NumPy array:
像排序过的购物清单一样， NumPy arrays 允许快速计算。数组可以快速访问元素。让我们创建一个简单的一维 NumPy 数组：

Python

CopyPlay

1# Creating a one-dimensional (1D) numpy array 2array_1d = np.array([1, 2, 3, 4, 5]) 3print(array_1d) # prints: [1 2 3 4 5]

This code creates a five-element array.
这段代码创建了一个包含五个元素的数组。

Creating Multi-Dimensional Arrays

创建多维数组

We can create multi-dimensional arrays as much as we would with a multi-day shopping list. Here, each sublist [] forms a row in the final array:
我们可以像创建多天的购物清单一样创建多维数组。在这里，每个子列表 [] 构成最终数组中的一行：

Python

CopyPlay

1# Two-dimensional (2D) numpy array 2array_2d = np.array([[1, 2, 3],[4, 5, 6]]) 3print(array_2d) 4'''prints: 5[[1 2 3] 6 [4 5 6]] 7'''

Each row in the output corresponds to a sublist in the input list.
输出中的每一行对应于输入列表中的一个子列表。

We can apply the same principle to create a three-dimensional array:
我们可以应用相同的原理来创建一个三维数组：

Python

CopyPlay

1# Three-dimensional (3D) numpy array 2array_3d = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]) 3print(array_3d) 4'''prints: 5[[[ 1 2 3] 6 [ 4 5 6]] 7 8 [[ 7 8 9] 9 [10 11 12]]] 10'''

Arrays Properties: Size 数组属性：大小

NumPy arrays come with a series of built-in properties that give helpful information about the structure and type of data they hold. These are accessible via the size, shape, and type fields, respectively.
NumPy 数组带有一系列内置属性，可提供有关其保存的数据结构和类型的有用信息。 可以分别通过  size, 、 shape  和  type  字段访问这些属性。

Let’s start with size. This property indicates the total number of elements in the array. Elements can be numbers, strings, etc. This becomes especially useful when working with multi-dimensional arrays where manually counting elements can be tedious.
我们先来看一下 size 。这个属性表示数组中元素的总数。元素可以是数字、字符串等等。在处理多维数组时，这个属性特别有用，因为手动计算元素数量可能很繁琐。

Python

CopyPlay

1array_3d = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]) 2print("Size:", array_3d.size) # Size: 12

The array above is a 3D array that contains two 2D arrays. Each of the 2D arrays has two arrays, and each of those has three elements. Therefore, the total number of elements is 2 * 2 * 3 = 12.
上面的数组是一个包含两个二维数组的三维数组。每个二维数组又包含两个数组，每个数组有三个元素。因此，元素总数为 2 * 2 * 3 = 12 。

Arrays Properties: Shape 数组属性：形状

Next, we have shape, which gives us the array’s dimensions. The shape property returns a tuple where the number of items is the dimension of the array, and the value of each item is the size of each dimension.
接下来是 shape ，它返回数组的维度。shape 属性返回一个元组，元组中元素的数量表示数组的维度数，每个元素的值表示对应维度的长度。

Python

CopyPlay

1print("Shape:", array_3d.shape) # Shape: (2, 2, 3)

In the example above, the shape (2, 2, 3) is a tuple of three values, which indicates that our array is 3D and contains two arrays, each of which includes two more arrays, each of which holds three elements.
在上面的例子中，形状 (2, 2, 3) 是一个包含三个值的元组，这表明我们的数组是三维的，它包含两个数组，每个数组又包含两个数组，每个数组又包含三个元素。

Arrays Properties: Type 数组属性：类型

Lastly is dtype, which stands for the data type. This property tells us about the elements stored in the array, whether they’re integers, floats, strings, etc.
最后是 dtype ，它代表数据类型。此属性告诉我们数组中存储的元素是什么类型，例如整数、浮点数、字符串等。

Python

CopyPlay

1print("Data type:", array_3d.dtype) # Data type: int64

For our example, the data type is int64 because our array only contains integers. If it had held floating point numbers, the dtype would have reflected that.
对于我们的示例，数据类型是 int64 ，因为我们的数组只包含整数。如果它包含浮点数，则 dtype 会反映这一点。

Understanding these properties is vital for effectively working with NumPy arrays, as they provide information about the array’s structure and content.
了解这些属性对于有效地使用 NumPy 数组至关重要，因为它们提供了有关数组结构和内容的信息。

Lesson Summary 课程总结

Great job! We have learned how to create basic and multi-dimensional NumPy arrays and examined the properties of arrays. Now, let’s move on to some exercises to practice these concepts and prepare for future data analysis challenges.
太棒了！我们已经学习了如何创建基本和多维 NumPy 数组，并检查了数组的属性。现在，让我们继续做一些练习来实践这些概念，并为未来的数据分析挑战做好准备。

Explore More of the 20 Newsgroups Dataset

Great job, Star Seeker! Now, let’s challenge you a bit more. We have the grid_power array, representing the power generated by a 1D line of solar panels in a space station. Your task is to modify the code to transform grid_power from a 1D numpy array to a 2D numpy array with 2 rows. Keep up the good work and let’s conquer the cosmos of knowledge!import numpy as np # Assume we have four solar panels grid_power = np.array([500, 700, 700, 900]) # Print the array print("\nSpaceship Grid Power:", grid_power) # Display the size, shape, and data type of our array print(“Size of Spaceship Grid Power Array:”, grid_power.size) print(“Shape of Spaceship Grid Power Array:”, grid_power.shape) print(“Data type of Spaceship Grid Power Array:”, grid_power.dtype)

Uncover the End of 20 Newsgroups Dataset

Great job, Star Seeker! Now, let’s challenge you a bit more. We have the grid_power array, representing the power generated by a 1D line of solar panels in a space station.

Your task is to modify the code to transform grid_power from a 1D numpy array to a 2D numpy array with 2 rows.

Keep up the good work and let’s conquer the cosmos of knowledge!

import numpy as np

  

# Assume we have four solar panels

grid_power = np.array([500, 700, 700, 900])

  

# Print the array

print("\nSpaceship Grid Power:", grid_power)

  

# Display the size, shape, and data type of our array

print("Size of Spaceship Grid Power Array:", grid_power.size)

print("Shape of Spaceship Grid Power Array:", grid_power.shape)

print("Data type of Spaceship Grid Power Array:", grid_power.dtype)

解题

import numpy as np

# Assume we have four solar panels
grid_power = np.array([500, 700, 700, 900])

# Reshape the array to 2D with 2 rows
grid_power_2D = grid_power.reshape(2, -1)

# Print the array
print("\nSpaceship Grid Power (2D):", grid_power_2D)

# Display the size, shape, and data type of our array
print("Size of Spaceship Grid Power Array (2D):", grid_power_2D.size)
print("Shape of Spaceship Grid Power Array (2D):", grid_power_2D.shape)
print("Data type of Spaceship Grid Power Array (2D):", grid_power_2D.dtype)

代码解释

The given code uses the NumPy library to transform a 1-dimensional (1D) numpy array, grid_power, into a 2-dimensional (2D) numpy array with 2 rows. The original grid_power array represents the power generated by a 1D line of solar panels in a space station.

Here’s a breakdown of the code:

1. The NumPy library is imported as np.
2. A 1D numpy array grid_power is created with four elements: [500, 700, 700, 900].
3. The reshape function is used to transform the 1D array into a 2D array with 2 rows. The -1 argument is used to automatically calculate the number of columns based on the size of the original array.
4. The transformed 2D array is printed using the print function.
5. The size, shape, and data type of the transformed 2D array are displayed using the size, shape, and dtype attributes of the numpy array.

The output of the code shows that the transformed 2D array is [[500, 700], [700, 900]], which means that the power generated by the first two solar panels is [500, 700] and the power generated by the last two solar panels is [700, 900]. The size of the transformed 2D array is 4, the shape is (2, 2), and the data type is int64.

给定的代码使用 NumPy 库将一个一维（1D）numpy 数组 grid_power 转换为一个具有 2 行的二维（2D）numpy 数组。原始的 grid_power 数组表示空间站中一排太阳能电池板生成的功率。

以下是代码的详细解释：

1. 导入 NumPy 库，并将其重命名为 np。
2. 创建一个具有四个元素的一维 numpy 数组 grid_power：[500, 700, 700, 900]。
3. 使用 print 函数打印原始的一维数组。
4. 使用 size、shape 和 dtype 属性分别显示原始一维数组的大小、形状和数据类型。
5. 使用 reshape 函数将一维数组转换为具有 2 行的二维数组。-1 参数用于根据原始数组的大小自动计算列数。
6. 使用 print 函数打印转换后的二维数组。
7. 使用 size、shape 和 dtype 属性分别显示转换后的二维数组的大小、形状和数据类型。

代码的输出显示，原始一维数组的大小为 4，形状为 (4,)，数据类型为 int64。将数组转换为具有 2 行的二维数组后，大小保持为 4，形状变为 (2, 2)，数据类型保持为 int64。转换后的二维数组为 [[500, 700], [700, 900]]，这意味着前两个太阳能电池板生成的功率为 [500, 700]，后两个太阳能电池板生成的功率为 [700, 900]。

Fetch Specific Categories from Dataset

Now, we need to make some modifications. In the code snippet provided, please add the necessary code to create a NumPy array that represents the power levels of spaceship engines. Feel free to select power levels that you deem appropriate. In addition, please insert the appropriate print statements to display the size, shape, and data type of this array.

May the force be with you!

import numpy as np

  

# TODO: Declare a numpy array for the power of spaceship engines. Choose any four powers you prefer.

# TODO: Print the created array.

  

# TODO: Display the size, shape, and data type of the created array.

解题

import numpy as np

# Declare a numpy array for the power of spaceship engines. Choosing four powers: 100, 200, 150, 300
engine_power = np.array([100, 200, 150, 300])

# Print the created array
print("Spaceship Engine Power:", engine_power)

# Display the size, shape, and data type of the created array
print("Size of Spaceship Engine Power Array:", engine_power.size)
print("Shape of Spaceship Engine Power Array:", engine_power.shape)
print("Data type of Spaceship Engine Power Array:", engine_power.dtype)

解释 The provided code is written in Python and uses the NumPy library to create and manipulate arrays. Here’s a step-by-step explanation of the code and the solution:

1. The NumPy library is imported using the import numpy as np statement.
2. A NumPy array named engine_power is declared to represent the power levels of spaceship engines. The power levels are chosen as [100, 200, 150, 300].
3. The created array is printed using the print function with a descriptive message.
4. The size, shape, and data type of the created array are displayed using the size, shape, and dtype attributes of the NumPy array.

The output of the code will be:

Spaceship Engine Power: [100 200 150 300] Size of Spaceship Engine Power Array: 4 Shape of Spaceship Engine Power Array: (4,) Data type of Spaceship Engine Power Array: int64

This output indicates that the engine_power array has a size of 4, a shape of (4,), and a data type of int64. This means that the array is a one-dimensional array with 4 elements, and each element is an integer. 提供的代码创建了一个名为 engine_power 的 NumPy 数组，用于表示太空飞船引擎的功率级别。选择的功率级别为 [100, 200, 150, 300]。然后，代码打印创建的数组并显示其大小、形状和数据类型。

Fetching the Third Article from Dataset

Excellent work so far! Now, let’s venture into the cosmos of data transformation.

Suppose you are given an array of temperatures in degrees Celsius. Your mission, should you choose to accept it, is to write a custom function to convert these temperature values from Celsius to Fahrenheit and then apply this function to the existing array.

Remember, the formula to convert Celsius to Fahrenheit is F=C∗1.8+32F=C∗1.8+32.

Prepare for liftoff!

import numpy as np

  

# Array of temperatures in Celsius for 5 consecutive days

temperatures_c = np.array([20.5, 25.3, 19.6, 22.7, 24.1])

  

# TODO: Write a function to convert temperatures from Celsius to Fahrenheit

  

# TODO: Apply the function to the temperatures_c array and transform them to Fahrenheit scale

  

print("Temperatures in Celsius: ", temperatures_c)

# TODO: Print the transformed Fahrenheit temperatures

Let’s proceed with the task of converting an array of temperatures from Celsius to Fahrenheit using a custom function. Here’s a step-by-step explanation of the process, followed by the actual implementation in Python:

1. Define the Conversion Function:

   • The conversion formula from Celsius to Fahrenheit is ( F = C \times 1.8 + 32 ).
   • We will define a function in Python to perform this calculation.

2. Apply the Function to the Array:

   • Use NumPy to handle the array of temperatures efficiently.
   • Apply the conversion function to each element in the array using vectorized operations.

Here’s the Python code to accomplish this:

import numpy as np

# Array of temperatures in Celsius for 5 consecutive days
temperatures_c = np.array([20.5, 25.3, 19.6, 22.7, 24.1])

# Function to convert temperatures from Celsius to Fahrenheit
def celsius_to_fahrenheit(celsius_temp):
    return celsius_temp * 1.8 + 32

# Apply the function to the temperatures_c array and transform them to Fahrenheit scale
temperatures_f = celsius_to_fahrenheit(temperatures_c)

# Print the original temperatures in Celsius
print("Temperatures in Celsius: ", temperatures_c)

# Print the transformed temperatures in Fahrenheit
print("Temperatures in Fahrenheit: ", temperatures_f)

Explanation of the Code:

• Importing NumPy: NumPy is imported for efficient array handling.
• Creating the Array: temperatures_c is defined as a NumPy array containing temperatures in Celsius.
• Defining the Conversion Function: The function celsius_to_fahrenheit takes a temperature in Celsius and converts it to Fahrenheit using the formula.
• Applying the Function: The function is applied to the entire array using NumPy’s vectorized operations, which allows for efficient computation.
• Printing Results: Finally, the original Celsius temperatures and the converted Fahrenheit temperatures are printed.

You can run this code in a Python environment to see the conversion results.

Exploring Text Length in Newsgroups Dataset

Great work, space explorer! You’ve traveled across galaxies of code and honed your skills in applying functions to arrays!

We’ve established a connection with a mysterious alien civilization. Interestingly, they value the squares of numbers and square roots of numbers more than the numbers itself! They provided us with a number sequence from 1 to 10 and seek the square and the square root of each number.

In this case, no need to vectorize custom function. We can use np.sqrt function for square roots and simply square the whole array with ** 2!

import numpy as np

  

# Array of numbers

numbers = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

  

# TODO: find squares of numbers

# TODO: find square roots of numbers

# TODO: print numbers, their squares and their square roots

Lesson 2: Mastering Text Cleaning for NLP: Techniques and Applications

Buckle up for today’s journey in the world of NumPy arrays! We’ll be exploring new routes in data selection. Isn’t that exciting? Ready to embark? 🚀

Lesson Overview and Plan

Hello, Data Whizz! Today’s lesson will focus on “selecting data” in NumPy arrays—our goal is to master integer and Boolean indexing and slicing. So, let’s jump in!

Understanding Indexing in 1D Arrays

In this section, we’ll discuss “Indexing”. It’s akin to item numbering in lists, and Python implements it with a zero-based index system. Let’s see this principle in action.

import numpy as np

# Let's form a 1D array and select, say, the 4th element
array_1d = np.array([2, 3, 5, 7, 11, 13, 17, 19, 23, 29])
fourth_element = array_1d[3]  # Selected element: 7

# To grab the last element, use a negative index
last_element = array_1d[-1]  # Selected element: 29

Essentially, indexing is a numeric system for items in an array—relatively straightforward!

Understanding Indexing in Multi-dimensional Arrays

Are you mindful of higher dimensions? NumPy arrays can range from 1D to N-dimensions. To access specific elements, we use the index pair (i,j) for a 2D array, (i,j,k) for a 3D array, and so forth.

# Form a 2D array and get the element at row 2 (indexed 1) and column 1 (indexed 0)
array_2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
element = array_2d[1,0]  # Selected element: 4

In this example, we selected the first element of the second row in a 2D-array—it’s pretty simple!

Understanding Boolean Indexing

Are you ready for some magic? Enter “Boolean Indexing”, which functions like a ‘Yes/No’ filter, with ‘Yes’ representing True and ‘No’ for False.

# A boolean index for even numbers
array_1d = np.array([8, 4, 7, 3, 4, 11])
even_index = array_1d % 2 == 0
even_numbers = array_1d[even_index]
print(even_numbers)  # Output: [8 4 4]

Or we can put the condition directly into [] brackets:

# A boolean index for even numbers
array_1d = np.array([8, 4, 7, 3, 4, 11])
even_numbers = array_1d[array_1d % 2 == 0]
print(even_numbers)  # Output: [8 4 4]

Voila! Now, we can filter data based on custom conditions.

Understanding Complex Conditions in Boolean Indexing

Now that we’ve mastered the simple ‘Yes/No’ binary filter system, let’s up the ante with “Complex Conditions in Boolean Indexing”. This method refines our filtering process further, allowing us to set more detailed restrictions.

Imagine, for instance, that we want to create an index for even numbers greater than five. We’ll merge two conditions to yield this result:

# A combined boolean index for even numbers > 5
array_1d = np.array([8, 4, 7, 3, 4, 11])
even_numbers_greater_than_five = array_1d[(array_1d % 2 == 0) & (array_1d > 5)]
print(even_numbers_greater_than_five)  # Output: [8]

In this query, we used the ampersand (&) to signify intersection - i.e., we’re selecting numbers that are both even AND larger than five. Note, that simple and operator won’t work here.

Similarly, we can use the pipe operator (|) to signify union - i.e., selecting numbers that are either even OR larger than five:

# A combined boolean index for even numbers or numbers > 5
array_1d = np.array([8, 4, 7, 3, 4, 11])
even_numbers_or_numbers_greater_than_five = array_1d[(array_1d % 2 == 0) | (array_1d > 5)]
print(even_numbers_or_numbers_greater_than_five)  # Output: [8 4 7 4 11]

Awesome, right? This additional filtering layer empowers us to be more specific and intentional about the data we select.

A Look at Slicing in NumPy Arrays

NumPy arrays could be sliced in the same manner as the regular python list. Let’s make a quick recall. The syntax is start:stop:step, where start is the first index to choose (inclusive), stop is the last index to choose (exclusive), and step defines the step of the selection. For example, if the step=1, each element will be selected, and if step=2 – every other one will be skipped. Let’s take a look at simple examples:

# Select elements at index 0, 1, 2
array_1d = np.array([1, 2, 3, 4, 5, 6])
first_three = array_1d[0:3]
print(first_three)  # Output: [1 2 3]

Note that slicing is inclusive on the left and exclusive on the right.

Another example with a step parameter:

# Select elements at odd indices 1, 3, 5, ...
array_1d = np.array([1, 2, 3, 4, 5, 6])
every_second = array_1d[1:6:2]
print(every_second)  # Output: [2 4 6]

In this case, we choose every second element, by starting with 1 and using step=2.

Lesson Summary

Congratulations! We’ve traversed the landscape of NumPy arrays, delving into indexing, Boolean indexing, and slicing. Now, we’ll dive into some hands-on exercises. After all, practice makes perfect. Let’s go forth, data explorers!

Update String and Clean Text

Are you ready to don your Data Analyst cape, Space Voyager? I have a task that will bring us down to Earth for a short while. Let’s envisage that you are a data analyst at an HR company, and your task is to filter out employees who earn more than an average wage of $30/hr.

This seems like quite a conundrum, doesn’t it? Don’t worry; we already have a solution.

Simply click Run to see how this task can be accomplished using NumPy's data selection techniques.

import numpy as np

  

# Create a NumPy array of wage per hour

wages_per_hour = np.array([14, 16, 20, 22, 25, 28, 30, 35, 38, 40, 45, 50])

  

# Select wages greater than 30 per hour

print(wages_per_hour[wages_per_hour > 30])

• 以下是对代码的逐步解释：

  1. 使用 import numpy as np 语句导入 NumPy 库。
  2. 创建一个名为 wages_per_hour 的 NumPy 数组，用于存储一组员工的每小时工资数据。
  3. 为了筛选出赚取超过 $30/hr 的员工，使用了布尔索引技术。表达式 wages_per_hour > 30 返回一个布尔数组，其中 True 对应于满足条件的索引（即工资超过 30）。
  4. 然后使用 print 函数打印筛选后的数组。

  代码的输出将是：

  [35 38 40 45 50]

  这个输出表明，工资为 $35/hr、$38/hr、$40/hr、$45/hr 和 $50/hr 的员工的工资超过了平均工资 $30/hr。

Filling in Python Functions and Regex Patterns

Well done, Space Voyager! You’ve harnessed the Power Cosmic to select adults from a group. Now, let’s make a slight alteration to the course.

Suppose you’re the coach of a team, which consists of both adults (aged 18 and above) and minors. The challenge lies in selecting only those adults who are under 30 to form the Junior Adults Team.

Trust your knowledge of python and apply it to the existing code. Are you ready? Set? Punch it to warp speed!

import numpy as np

  

# Create a NumPy array of ages

ages = np.array([15, 18, 16, 19, 36, 34, 27, 21, 23, 25])

  

# Select adults age under 30 years

adults = ages[ages > 18]

print("Adults:", adults)

以下是对代码的逐步解释：

1. 使用 import numpy as np 语句导入 NumPy 库。
2. 创建一个名为 ages 的 NumPy 数组，用于存储一组个体的年龄。
3. 为了选择成年人（年龄18岁及以上），使用了布尔索引技术。表达式 ages > 18 返回一个布尔数组，其中 True 对应于满足条件的索引（即年龄大于18）。
4. 结果数组的成年人存储在名为 adults 的变量中。
5. 为了仅选择那些年龄小于30岁的成年人，在 adults 数组上使用了另一种布尔索引技术。表达式 adults < 30 返回一个布尔数组，其中 True 对应于满足条件的索引（即年龄小于30）。
6. 然后使用 print 函数打印筛选后的年龄小于30岁的成年人数组。

代码的输出将是：

成年人：[19 36 34 27 21 23 25] 青年成年人：[19 27 21 23 25]

这个输出表明，该组中的成年人年龄为19、36、34、27、21、23和25岁。在这些成年人中，年龄小于30岁的人年龄为19、27、21、23和25岁。

Of course, I’d be happy to help with that! To modify the code to select adults under 30, you can add an additional condition to the Boolean indexing expression. Here’s a hint on how to proceed:

1. You already have the Boolean indexing expression ages > 18 to select adults.
2. To limit the age to under 30, you can add another condition using the less than operator <.
3. Use the & operator to combine the two conditions.
4. Apply the combined condition to the ages array to select adults under 30.
5. Print the filtered array to verify that it contains the correct results.

Here’s the modified code:

import numpy as np

# Create a NumPy array of ages
ages = np.array([15, 18, 16, 19, 36, 34, 27, 21, 23, 25])

# Select adults under 30 years
adults_under_30 = ages[(ages > 18) & (ages < 30)]

print("Adults under 30:", adults_under_30)

This code will select adults (ages 18 and above) who are under 30 years old and print the filtered array.

Mastering Text Cleaning with Python Regex

Good try but that’s not exactly right.

Your solution currently selects adults over 18 years but does not limit the age to under 30.

Do you need a hint on how to modify it to meet the requirement? 🤔

import numpy as np

  

# Create a NumPy array of ages

ages = np.array([15, 18, 16, 19, 36, 34, 27, 21, 23, 25])

  

# Select adults age under 30 years

adults = ages[ages > 18]

print("Adults:", adults)

Sure, let’s modify the existing code to select only those adults who are under 30 years of age. Here is the updated code:

import numpy as np

# Create a NumPy array of ages
ages = np.array([15, 18, 16, 19, 36, 34, 27, 21, 23, 25])

# Select adults aged 18 and above, but under 30 years
junior_adults = ages[(ages >= 18) & (ages < 30)]
print("Junior Adults:", junior_adults)

In this code, we use a boolean condition to select ages that are both 18 and above but less than 30. Running this will produce the output:

Junior Adults: [19 27 21 23 25]

This way, only the adults who are under 30 are selected for the Junior Adults Team.

Implement Text Cleaning on Dataset

Great job, Space Explorer! We’re planning to embark on a more advanced extraction mission. Imagine we are choosing a crew of space companions – cool space dogs, just like me! Within the field of weight analysis, envision a scenario where we’re tasked to identify dogs whose weight exceeds 20 kg or is under 10 kg. The information will enable us to concentrate our study on this group and assess their potential health risks.
太空探险家，干得好！我们正计划开展一项更先进的提取任务。想象一下，我们正在挑选太空伙伴——就像我一样的酷酷的太空狗！在体重分析领域，设想一个场景，我们需要识别体重超过 20 kg 或低于 10 kg 的狗。这些信息将使我们能够将研究集中在这个群体上，并评估它们的潜在健康风险。

For boolean selection with AND operand, we used the & operator. For the OR operand, use a vertical line |!
对于使用 AND 运算符的布尔选择，我们使用 & 运算符。对于 OR 运算符，请使用竖线 | ！

Are you ready to code? Let’s get started!
准备开始编程了吗？我们开始吧！

import numpy as np

  

# Create a NumPy array

weight = np.array([12, 15, 11, 9, 23, 10, 12, 20, 25, 25, 8, 15, 4])

  

# TODO: Select weights greater than 20 or less than 10

解释 Absolutely! Let’s identify the space dogs whose weight exceeds 20 kg or is under 10 kg using the | operator for the OR condition. Here is the updated code:

import numpy as np

# Create a NumPy array
weight = np.array([12, 15, 11, 9, 23, 10, 12, 20, 25, 25, 8, 15, 4])

# Select weights greater than 20 or less than 10
selected_weights = weight[(weight > 20) | (weight < 10)]
print("Selected Weights:", selected_weights)

Running this code will produce the output:

Selected Weights: [ 9 23 25 25  8  4]

This code identifies and prints the weights of the space dogs that are either over 20 kg or under 10 kg, helping you to focus your study on this group for potential health risk assessments.

Mastering Text Cleaning with Python Regex on a Dataset

Bravo, fellow space explorer! We’ve unraveled the mysteries of data selection in NumPy arrays. Now, in this quadrant’s final test, let’s imagine that you are a space health officer.

You have an array of crew members’ badge numbers and need to choose some subgroup of people for a health check-up. Write a code that selects crew members with even badge number lower than 33. After this, select the first three crew members from the obtained list, using array slicing. Remember, the syntax is [start:stop:step], just like with python lists!
你有一个数组，存储着船员的徽章号码，你需要从中选择一些人进行健康检查。编写代码，选择徽章号码为偶数且小于  33  的船员。然后，使用数组切片从获得的列表中选择前三名船员。记住，语法是  [start:stop:step]  ，就像 Python 列表一样！

Let’s rev up our coding hyperdrive!
让我们加速代码引擎！

import numpy as np

  

badges = np.array([12, 14, 15, 20, 32, 33, 40, 42])

  

# TODO: Using a condition-based selection, filter badges that has even number lower than 33

  

# TODO: Print the filtered numbers.

解释 Let’s embark on this final mission. We’ll filter the crew members with even badge numbers lower than 33 and then select the first three members from the resulting list using array slicing. Here is the code to achieve this:

import numpy as np

# Array of crew members' badge numbers
badges = np.array([12, 14, 15, 20, 32, 33, 40, 42])

# Filter badges that have even numbers and are lower than 33
filtered_badges = badges[(badges % 2 == 0) & (badges < 33)]

# Select the first three crew members from the filtered list
selected_crew = filtered_badges[:3]

# Print the filtered numbers
print("Filtered Badges:", filtered_badges)
print("Selected Crew for Health Check-up:", selected_crew)

Running this code will produce the following output:

Filtered Badges: [12 14 20 32]
Selected Crew for Health Check-up: [12 14 20]

This code successfully filters the badge numbers to find those that are even and less than 33, and then selects the first three members from this filtered list for the health check-up.

Lesson 3: Removing Stop Words and Stemming in Text Preprocessing

Introduction and Lesson Overview

Welcome! In today’s lesson, we’ll learn how to use functions with NumPy arrays, a crucial concept for managing large amounts of data. This ability facilitates efficient data processing — a teacher calculating class averages or a business analyst summarising company sales illustrates the practical use of these skills. Are you ready? Let’s get started! 欢迎！在今天的课程中，我们将学习如何对 NumPy 数组使用函数，这对管理大量数据至关重要。这种能力有助于高效的数据处理——教师计算班级平均成绩或业务分析师汇总公司销售额，都体现了这些技能的实际应用。准备好了吗？让我们开始吧！

Arithmetic Operations with NumPy Arrays

NumPy 数组的算术运算

Two arrays of the same shape can undergo basic arithmetic operations. The operations are performed element-wise, meaning they’re applied to each pair of corresponding elements. Suppose we have two grade arrays of students from two subjects. By adding these arrays, we can calculate the total grades:
形状相同的两个数组可以进行基本的算术运算。这些运算以元素方式执行，这意味着它们应用于每对对应的元素。假设我们有两个科目学生的成绩数组。通过将这些数组相加，我们可以计算总成绩：

subject1_grades = np.array([88, 90, 75, 92, 85])
subject2_grades = np.array([92, 85, 78, 90, 88])

total_grades = subject1_grades + subject2_grades

print("Total grades:", total_grades)  # Output: Total grades: [180 175 153 182 173]

The two arrays are added element-wise in this code to calculate the total grades.
在这段代码中，两个数组按元素相加来计算总成绩。

Introduction to Universal Functions (ufuncs)

通用函数 (ufuncs) 简介

NumPy’s Universal Functions (also called ufuncs) perform element-wise operations on arrays, including mathematical functions like sin, cos, log, and sqrt. Let’s look at a use case:
NumPy 的通用函数（也称为 ufuncs ）对数组执行元素级操作，包括 sin 、 cos 、 log 和 sqrt 等数学函数。我们来看一个用例：

angles_degrees = np.array([0, 30, 45, 60, 90])
angles_radians = np.radians(angles_degrees)
sin_values = np.sin(angles_radians)

print("Sine of angles:", sin_values)  # Output: Sine of angles: [0. 0.5 0.70710678 0.8660254 1.]

This code applies np.sin universal function to each array element. As np.sin expects its input in radians, we first transform degrees to radians with np.radians, applying it to each array element similarly.
这段代码对每个数组元素应用 np.sin 通用函数。由于 np.sin 要求输入为弧度，我们首先使用 np.radians 将角度转换为弧度，并同样将其应用于每个数组元素。

Extending NumPy with Custom Functions

使用自定义函数扩展 NumPy 功能

NumPy allows the application of a custom function to each element of the array separately by transforming the target function with np.vectorize. Let’s create a function to check the parity of a number:
NumPy 允许通过使用 np.vectorize 转换目标函数，将自定义函数分别应用于数组的每个元素。让我们创建一个函数来检查数字的奇偶性：

def is_even(n):
    return n % 2 == 0

vectorized_is_even = np.vectorize(is_even)

numbers = np.array([1, 2, 3, 4, 5])
results = vectorized_is_even(numbers)

print("Results:", results).  # Output: Results: [False True False True False]

The vectorized_is_even function returns an array indicating whether each value in numbers is even.
vectorized_is_even 函数返回一个数组，指示 numbers 中的每个值是否为偶数。

Lesson Summary 课程总结

Well done! You’ve learned how to apply functions to NumPy arrays, perform arithmetic operations, apply statistical functions, use ufuncs, and extend NumPy with custom functions. Up next are practice exercises for further learning. Let’s proceed!
干得好！您已经学习了如何将函数应用于 NumPy 数组、执行算术运算、应用统计函数、使用 ufuncs 以及使用自定义函数扩展 NumPy。接下来是用于进一步学习的练习。让我们继续吧！

「Pratice」Switch from LancasterStemmer to PorterStemmer

Hello there, space explorer! Are you ready to venture further into the mysterious universe of data analysis with Python and NumPy? Let’s do this!

Imagine that you’ve just hosted a space-themed quiz in your galactic neighborhood. Now, you have the raw scores of the participants. However, the participants are eager to know their percentage scores.

Luckily, all the work is done! Simply press the Run button to see how we’ve calculated the percentage scores using a function applied to a NumPy array.

import numpy as np

  

# Function to calculate percentage marks

def percentage_calc(percentage_marks):

return (percentage_marks * 100) / 75

  

# Array of raw marks

raw_marks = np.array([35, 48, 60, 72, 75])

  

# Compute percentage using our function

percentage_calc = np.vectorize(percentage_calc)

percentage_marks = percentage_calc(raw_marks)

  

print("Raw Marks: ", raw_marks)

print("Percentage Marks: ", percentage_marks)

解题

import numpy as np

  

# Function to calculate percentage marks

def percentage_calc(percentage_marks):

  return (percentage_marks * 100) / 75

  

# Array of raw marks

raw_marks = np.array([35, 48, 60, 72, 75])

  

# Compute percentage using our function

percentage_calc = np.vectorize(percentage_calc)

percentage_marks = percentage_calc(raw_marks)

  

print("Raw Marks: ", raw_marks)

print("Percentage Marks: ", percentage_marks)

output

Raw Marks:  [35 48 60 72 75]
Percentage Marks:  [46.66666667 64.         80.         96.         100.        ]

This code snippet demonstrates a simple yet effective way to calculate percentage scores from raw marks using NumPy. Here’s a breakdown:

1. Importing NumPy: import numpy as np imports the NumPy library, essential for array operations.

2. Defining the percentage_calc function: This function takes a single argument, percentage_marks, representing the raw mark, and calculates the corresponding percentage score. The formula used is (percentage_marks * 100) / 75, where 75 is assumed to be the total marks for the quiz.

3. Creating the raw_marks array: raw_marks = np.array([35, 48, 60, 72, 75]) defines a NumPy array containing the raw scores of the participants.

4. Vectorizing the percentage_calc function: percentage_calc = np.vectorize(percentage_calc) converts the percentage_calc function into a vectorized function. This allows it to operate on each element of the raw_marks array individually.

5. Calculating percentage marks: percentage_marks = percentage_calc(raw_marks) applies the vectorized percentage_calc function to the raw_marks array, resulting in a new array percentage_marks containing the calculated percentage scores.

6. Printing the results: Finally, print("Raw Marks: ", raw_marks) and print("Percentage Marks: ", percentage_marks) display the original raw marks and the calculated percentage scores.

This code snippet showcases the power of NumPy for efficiently performing calculations on arrays. The vectorization technique allows for concise and efficient code, making it ideal for data analysis tasks.

「Pratice」 Removing Stop Words and Punctuation from Text

Congratulations on successfully running your first code! But, what else can you accomplish? Let’s investigate!

The provided starter code applies a 20% discount to all prices. However, what if the store policy were to change, offering a 30% discount for prices greater than 350? For other prices, discount stays the same – 20%. Could you modify our code to reflect this new discount rate in our discounted prices?

It’s time to ignite those rockets!

import numpy as np

  

# Function to calculate the discount price

def discount_price(price):

return price - (price * 0.20) # 20% discount

  

# Array of original prices

prices = np.array([100, 200, 300, 400, 500])

  

# Compute discounted prices using our function

discount_price = np.vectorize(discount_price)

discounted_prices = discount_price(prices)

  

print("Original Prices: ", prices)

print("Discounted Prices : ", discounted_prices)

解题

import numpy as np

  

# Function to calculate the discount price

def discount_price(price):

  if price > 350:

    return price - (price * 0.30) # 30% discount

  else:

    return price - (price * 0.20) # 20% discount

  

# Array of original prices

prices = np.array([100, 200, 300, 400, 500])

  

# Compute discounted prices using our function

discount_price = np.vectorize(discount_price)

discounted_prices = discount_price(prices)

  

print("Original Prices: ", prices)

print("Discounted Prices : ", discounted_prices)

output输出：

Original Prices:  [100 200 300 400 500]
Discounted Prices :  [ 80. 160. 240. 280. 350.]

Here’s how the code has been modified to implement the new discount policy:

1. Conditional Discount in discount_price Function:

   • Inside the discount_price function, an if statement is added to check if the price is greater than 350.
   • If price > 350, a 30% discount is applied: return price - (price * 0.30).
   • Otherwise (if price <= 350), the original 20% discount is applied: return price - (price * 0.20).

2. Rest of the Code Remains the Same:

   • The code for creating the prices array, vectorizing the discount_price function, and calculating discounted_prices remains unchanged.

Now, the code correctly applies a 30% discount for prices above 350 and a 20% discount for prices below or equal to 350. This demonstrates the flexibility of using conditional logic within functions to handle varying discount scenarios.

「Pratice」Stemming Words with PorterStemmer

「Pratice」Implementing Stopword Removal and Stemming Function

Excellent work so far! Now, let’s venture into the cosmos of data transformation.

Suppose you are given an array of temperatures in degrees Celsius. Your mission, should you choose to accept it, is to write a custom function to convert these temperature values from Celsius to Fahrenheit and then apply this function to the existing array.

Remember, the formula to convert Celsius to Fahrenheit is F

C ∗ 1.8 + 32 F=C∗1.8+32.

Prepare for lift off!

import numpy as np

# Array of temperatures in Celsius for 5 consecutive days
temperatures_c = np.array([20.5, 25.3, 19.6, 22.7, 24.1])

# TODO: Write a function to convert temperatures from Celsius to Fahrenheit

# TODO: Apply the function to the temperatures_c array and transform them to Fahrenheit scale

print("Temperatures in Celsius: ", temperatures_c)
# TODO: Print the transformed Fahrenheit temperatures

Let’s proceed with the task of converting an array of temperatures from Celsius to Fahrenheit using a custom function. Here’s a step-by-step explanation of the process, followed by the actual implementation in Python:

1. Define the Conversion Function:

   • The conversion formula from Celsius to Fahrenheit is ( F = C \times 1.8 + 32 ).
   • We will define a function in Python to perform this calculation.

2. Apply the Function to the Array:

   • Use NumPy to handle the array of temperatures efficiently.
   • Apply the conversion function to each element in the array using vectorized operations.

Here’s the Python code to accomplish this:

import numpy as np

# Array of temperatures in Celsius for 5 consecutive days
temperatures_c = np.array([20.5, 25.3, 19.6, 22.7, 24.1])

# Function to convert temperatures from Celsius to Fahrenheit
def celsius_to_fahrenheit(celsius_temp):
    return celsius_temp * 1.8 + 32

# Apply the function to the temperatures_c array and transform them to Fahrenheit scale
temperatures_f = celsius_to_fahrenheit(temperatures_c)

# Print the original temperatures in Celsius
print("Temperatures in Celsius: ", temperatures_c)

# Print the transformed temperatures in Fahrenheit
print("Temperatures in Fahrenheit: ", temperatures_f)

Explanation of the Code:

• Importing NumPy: NumPy is imported for efficient array handling.
• Creating the Array: temperatures_c is defined as a NumPy array containing temperatures in Celsius.
• Defining the Conversion Function: The function celsius_to_fahrenheit takes a temperature in Celsius and converts it to Fahrenheit using the formula.
• Applying the Function: The function is applied to the entire array using NumPy’s vectorized operations, which allows for efficient computation.
• Printing Results: Finally, the original Celsius temperatures and the converted Fahrenheit temperatures are printed.

You can run this code in a Python environment to see the conversion results.

「Pratice」Cleaning and Processing the First Newsgroup Article

Great work, space explorer! You’ve traveled across galaxies of code and honed your skills in applying functions to arrays!

We’ve established a connection with a mysterious alien civilization. Interestingly, they value the squares of numbers and square roots of numbers more than the numbers itself! They provided us with a number sequence from 1 to 10 and seek the square and the square root of each number.

In this case, no need to vectorize custom function. We can use np.sqrt function for square roots and simply square the whole array with ** 2!

  

import numpy as np

  

  

# Array of numbers

  

numbers = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

  

  

# TODO: find squares of numbers

  

# TODO: find square roots of numbers

  

# TODO: print numbers, their squares and their square roots

  

Sure! Let’s proceed with finding the squares and square roots of the numbers from 1 to 10. We’ll use NumPy’s sqrt function for the square roots and the ** 2 operator for squaring the numbers. Here’s the complete Python code:

import numpy as np

# Array of numbers
numbers = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

# Find squares of numbers
squares = numbers ** 2

# Find square roots of numbers
square_roots = np.sqrt(numbers)

# Print numbers, their squares, and their square roots
print("Numbers: ", numbers)
print("Squares: ", squares)
print("Square Roots: ", square_roots)

Explanation:

• Array of Numbers: The array numbers contains the sequence from 1 to 10.
• Squares: Using the ** 2 operator, we compute the square of each number in the array.
• Square Roots: Using the np.sqrt function, we compute the square root of each number in the array.
• Printing Results: Finally, the original numbers, their squares, and their square roots are printed.

You can run this code in a Python environment to see the results. Here’s what the output should look like:

Numbers:  [ 1  2  3  4  5  6  7  8  9 10]
Squares:  [  1   4   9  16  25  36  49  64  81 100]
Square Roots:  [1.         1.41421356 1.73205081 2.         2.23606798 2.44948974
 2.64575131 2.82842712 3.         3.16227766]

Lesson 4: Unleashing the Power of n-grams in Text Classification

Lesson Overview 课程概述

Hello, students! Today, we’re exploring the pandas DataFrame, a powerhouse structure in data analysis with Python. We’ll contrast it with NumPy arrays and teach you how to build a DataFrame. Additionally, we’ll delve into its integral parts and data types.
同学们，大家好！今天，我们将探索 Pandas DataFrame，它是 Python 数据分析中的强大结构。我们会将其与 NumPy 数组进行对比，并教您如何构建 DataFrame。此外，我们还将深入研究其组成部分和数据类型。

Introduction to the Pandas Library

Pandas 库简介

The pandas library is Python’s solution for tabular data operations, packing more punch for data analysis than NumPy, which is skewed towards numerical computations. Pandas houses two fundamental data structures: the Series (1D) and the DataFrame (2D). Often, the DataFrame is the go-to choice. Let’s start by importing pandas: Pandas 库是 Python 用于表格数据操作的解决方案，它比偏向于数值计算的 NumPy 库在数据分析方面功能更强大。Pandas 包含两种基本数据结构： Series （一维）和 DataFrame （二维）。通常， DataFrame 是首选。让我们从导入 pandas 开始：

import pandas as pd

Here, pd serves as a standard alias for pandas.
这里， pd 是 pandas 的标准别名。

Creating a DataFrame 创建数据帧

Building a DataFrame in pandas is straightforward. It can be created from a dictionary, list, or NumPy array. Here’s an example of creating a DataFrame from a dictionary:
在 pandas 中构建 DataFrame 非常简单。它可以从字典、列表或 NumPy 数组创建。以下是如何从字典创建 DataFrame 的示例：

student_data = {  
    'Name': ['Sara', 'Ray', 'John'],
    'Age': [15, 16, 17],
}

df = pd.DataFrame(student_data)

print(df)

'''Output:
   Name  Age
0  Sara   15
1   Ray   16
2  John   17
'''

In the student_data dictionary, each (key, value) pair becomes a DataFrame column. The DataFrame automatically assigns an index (0-2) to each row, but we can also specify our own if we choose to do so.
在 student_data 字典中，每个 (键, 值) 对都会成为 DataFrame 的一列。DataFrame 会自动为每一行分配一个索引 (0-2)，但我们也可以选择指定自己的索引。

Adding a Column to the DataFrame

为数据帧添加列

Adding a new column to an existing DataFrame is straightforward: just like with dictionary, simply refer to a new key and assign an array to it. Let’s add information about student’s grades to our data:
向现有 DataFrame 添加新列很简单：就像使用字典一样，只需引用一个新键并为其分配一个数组即可。让我们将有关学生成绩的信息添加到我们的数据中：

df['Grade'] = [9, 10, 7]

print(df)

'''Output:
   Name  Age  Grade
0  Sara   15      9
1   Ray   16     10
2  John   17      7
'''

The 'Grade' column has been successfully added to the DataFrame.
'Grade' 列已成功添加到 DataFrame 中。

Classification of DataFrame Parts

DataFrame 部分的分类

A DataFrame consists of three components: data, index, and columns. Pandas neatly organize data into rows and columns — each row is a record, and each column is a feature (such as Name, Age, or Grade). The index, much like the index of a book, aids in data location. Columns serve as labels for the features in our data. Let’s analyze these components with our student data:
DataFrame 由三个部分组成：数据、索引和列。Pandas 将数据整齐地组织成行和列——每一行都是一条记录，每一列都是一个特征（例如姓名、年龄或年级）。索引就像书的索引一样，有助于数据定位。列充当数据中特征的标签。让我们用学生数据来分析这些组成部分：

print(df.index)  # Output: RangeIndex(start=0, stop=3, step=1)
print(df.columns)  # Output: Index(['Name', 'Age', 'Grade'], dtype='object') 

Our dataset contains row labels (index) from 0 to 2 and column labels (columns) of ‘Name’, ‘Age’, and ‘Grade’.
我们的数据集包含从 0 到 2 的行标签（索引）以及“姓名”、“年龄”和“年级”的列标签（列）。

Accessing DataFrame Columns

访问 DataFrame 列

One of the useful features of pandas DataFrame is its flexibility in accessing individual columns of data. You can select a single column, much like how you access a dictionary item by its key. Let’s use our student_data DataFrame again:
pandas DataFrame 的一个有用特性是它在访问单个数据列方面的灵活性。您可以选择单个列，就像通过键访问字典项一样。让我们再次使用我们的 student_data DataFrame：

print(df['Name'])

'''Output:
0    Sara
1     Ray
2    John
Name: Name, dtype: object
'''

As you can see, we now only have the ‘Name’ column content. Notice that pandas kept the index intact to retain information about the original rows.
如您所见，我们现在只有“姓名”列的内容。请注意，pandas 保留了索引不变，以保留有关原始行的信息。

To access multiple columns at once, pass a list with column names:
要访问多个列，请传递一个包含列名的列表：

print(df[['Name', 'Grade']])

'''Output:
   Name  Grade
0  Sara      9
1   Ray     10
2  John     11
'''

Adding a list of columns as input, we obtain a new DataFrame that only includes the students’ ‘Name’ and ‘Grade’.
输入列名列表，我们可以得到一个新的 DataFrame，其中仅包含学生的“姓名”和“年级”。

Remember: when handling a single column, pandas will return a Series. When dealing with multiple columns, pandas will return a DataFrame. Any new DataFrame or Series changes will not affect the original data.
记住：处理单列时，pandas 将返回 Series。处理多列时，pandas 将返回 DataFrame。任何新的 DataFrame 或 Series 更改都不会影响原始数据。

Understanding Data Types 理解数据类型

The strength of a DataFrame lies in its ability to accommodate multiple data types. Using the DataFrame.dtypes property, we can ascertain the data type of each column. Let’s look at the data types of out DataFrame:
DataFrame 的优势在于它能够适应多种数据类型。使用 DataFrame.dtypes 属性，我们可以确定每一列的数据类型。让我们看一下 DataFrame 的数据类型：

print(df.dtypes)

Name     object
Age       int64
Grade     int64
dtype: object

Note, that strings are represented as the object data type.
注意，字符串以 object 数据类型表示。

Unlike NumPy arrays, which harbor single-type data, pandas DataFrames can handle and manipulate different data types.
与只能处理单一数据类型的 NumPy arrays 不同，pandas DataFrames 可以处理和操作多种数据类型。

Lesson Summary 课程总结

And there we have it! We’ve traversed the fascinating terrain of the pandas DataFrame. We’ve compared it with NumPy arrays, learned how to build a DataFrame, and explored its structure and diverse data types. Now, it’s time to shift gears to some hands-on practice. You’ll master the art of creating pandas’ DataFrame, solidifying your understanding and bolstering your confidence.
好了！我们已经穿越了 pandas DataFrame 的奇妙世界。我们将它与 NumPy arrays 进行了比较，学习了如何构建 DataFrame ，并探索了它的结构和多样化的数据类型。现在，让我们开始实践吧。您将掌握创建 pandas DataFrame, 的技巧，巩固您的理解并增强您的信心。

「Pratice」Generating Bigrams and Trigrams with NLP

Astronauts, let’s beam the classroom data into our trusty spaceship, the pandas DataFrame! This handy tool will help us organize and analyze the students’ test scores for different subjects. Click on the Run button to see just how easy it is!
宇航员们，让我们将课堂数据传送进我们可靠的宇宙飞船“熊猫号”里。

import pandas as pd

  

classroom_data = {

'StudentName': ['Amy', 'Bob', 'Eva', 'Jake'],

'Subject': ['Math', 'English', 'Science', 'Arts'],

'TestScore': [89, 93, 91, 97]

}

  

classroom_df = pd.DataFrame(classroom_data)

print("\nClassroom DataFrame:\n", classroom_df)

  

classroom_df['ParticipationScore'] = [9, 10, 8, 10]

print("\n Updated DataFrame with 'ParticipationScore':\n", classroom_df)

「Pratice」 Generating Bigrams and Trigrams from Text Data

Absolutely! Let’s organize and analyze the classroom data using a pandas DataFrame. Here’s the complete Python code for creating the DataFrame, adding a new column for participation scores, and displaying the updated DataFrame:

import pandas as pd

# Classroom data
classroom_data = {
    'StudentName': ['Amy', 'Bob', 'Eva', 'Jake'],
    'Subject': ['Math', 'English', 'Science', 'Arts'],
    'TestScore': [89, 93, 91, 97]
}

# Create DataFrame
classroom_df = pd.DataFrame(classroom_data)
print("\nClassroom DataFrame:\n", classroom_df)

# Add ParticipationScore column
classroom_df['ParticipationScore'] = [9, 10, 8, 10]
print("\nUpdated DataFrame with 'ParticipationScore':\n", classroom_df)

Explanation:

• Import pandas: First, we import the pandas library.
• Define Classroom Data: The classroom data is defined as a dictionary containing student names, subjects, and test scores.
• Create DataFrame: We create a DataFrame from the classroom data using pd.DataFrame.
• Print Initial DataFrame: The initial DataFrame is printed to show the classroom data.
• Add Participation Score: A new column ParticipationScore is added to the DataFrame.
• Print Updated DataFrame: The updated DataFrame is printed to show the classroom data along with the participation scores.

Running this code will produce the following output:

Classroom DataFrame:
   StudentName  Subject  TestScore
0         Amy     Math         89
1         Bob  English         93
2         Eva  Science         91
3        Jake     Arts         97

Updated DataFrame with 'ParticipationScore':
   StudentName  Subject  TestScore  ParticipationScore
0         Amy     Math         89                  9
1         Bob  English         93                 10
2         Eva  Science         91                  8
3        Jake     Arts         97                 10

This demonstrates how easy it is to organize and analyze data using pandas DataFrame.

「Pratice」Generating Bigrams and Trigrams from Two Texts

Astronauts, let’s beam the classroom data into our trusty spaceship, the pandas DataFrame! This handy tool will help us organize and analyze the students’ test scores for different subjects. Click on the Run button to see just how easy it is!

import pandas as pd

  

classroom_data = {

'StudentName': ['Amy', 'Bob', 'Eva', 'Jake'],

'Subject': ['Math', 'English', 'Science', 'Arts'],

'TestScore': [89, 93, 91, 97]

}

  

classroom_df = pd.DataFrame(classroom_data)

print("\nClassroom DataFrame:\n", classroom_df)

  

classroom_df['ParticipationScore'] = [9, 10, 8, 10]

print("\n Updated DataFrame with 'ParticipationScore':\n", classroom_df)

输出

Classroom DataFrame:  
   StudentName  Subject  TestScore  
0         Amy     Math         89  
1         Bob  English         93  
2         Eva  Science         91  
3        Jake     Arts         97  
  
 Updated DataFrame with 'ParticipationScore':  
   StudentName  Subject  TestScore  ParticipationScore  
0         Amy     Math         89                   9  
1         Bob  English         93                  10  
2         Eva  Science         91                   8  
3        Jake     Arts         97                  10

解释 You’re right! This code snippet is a great introduction to using pandas DataFrames for organizing and analyzing data, especially in a space exploration context. Let’s break it down step by step:

1. Importing pandas:

import pandas as pd

• This line imports the pandas library, which is a powerful tool for data manipulation and analysis in Python. We use the as pd part to give the library a shorter alias, making it easier to use later.

2. Creating the Data Dictionary:

classroom_data = { 'StudentName': ['Amy', 'Bob', 'Eva', 'Jake'], 'Subject': ['Math', 'English', 'Science', 'Arts'], 'TestScore': [89, 93, 91, 97] }

• This creates a dictionary called classroom_data to store the information about the students.
• The dictionary has three keys:
  • StudentName: A list of student names.
  • Subject: A list of subjects.
  • TestScore: A list of test scores.

3. Creating the DataFrame:

classroom_df = pd.DataFrame(classroom_data)

• This line creates a pandas DataFrame called classroom_df from the classroom_data dictionary.
• DataFrames are like tables in spreadsheets, providing a structured way to organize and work with data.

4. Printing the DataFrame:

print("\nClassroom DataFrame:\n", classroom_df)

• This line prints the classroom_df DataFrame to the console. You’ll see a neatly formatted table with the student names, subjects, and test scores.

5. Adding a New Column:

classroom_df['ParticipationScore'] = [9, 10, 8, 10]

• This line adds a new column named ParticipationScore to the classroom_df DataFrame.
• It assigns the values [9, 10, 8, 10] to the new column, representing the participation scores for each student.

6. Printing the Updated DataFrame:

print("\n Updated DataFrame with 'ParticipationScore':\n", classroom_df)

• This line prints the updated classroom_df DataFrame, now including the ParticipationScore column.

In Summary:

This code demonstrates how to:

• Import the pandas library.
• Create a dictionary to hold data.
• Convert the dictionary into a pandas DataFrame.
• Print the DataFrame.
• Add a new column to the DataFrame.

This is a basic example of how pandas can be used to work with data in a structured and efficient way. You can use pandas to perform various operations on DataFrames, such as:

• Filtering: Selecting specific rows based on conditions.
• Sorting: Arranging data in ascending or descending order.
• Aggregation: Calculating summary statistics (like mean, median, etc.).
• Joining: Combining data from multiple DataFrames.

pandas is a powerful tool for data analysis in Python, and it’s widely used in various fields, including data science, machine learning, and finance.

「Pratice」Creating Bigrams from Preprocessed Text Data

Well done, Voyager! Now, let’s tweak the code a little. Your task is to modify it to add a new column, "Experience", to the teachers_df DataFrame. For 'Experience', use the list [12, 6, 15]. Reach for the stars!

import pandas as pd

  

teachers_data = {

'Name': ['Sarah', 'Mike', 'Emma'],

'Subject': ['English', 'Math', 'Science'],

}

  

teachers_df = pd.DataFrame(teachers_data)

print("Initial DataFrame:\n", teachers_df)

  

teachers_df['Years'] = [10, 5, 8]

print("\nDataFrame after a new column:\n", teachers_df)

「Pratice」Unigrams and Bigrams from Clean 20 Newsgroups Dataset

Lesson 5: Understanding Named Entity Recognition in NLP

Adding a Row to a DataFrame

Multiple scenarios might necessitate adding new entries to our DataFrame. Let’s explore how to accomplish that:

In modern pandas, we use pd.concat() function to incorporate new rows. If you forgot to add 'Pears' to your grocery list, here’s how to do it:

new_row = pd.DataFrame({'Grocery Item': ['Pears'], 'Price per kg': [4.00]})

grocery_df = pd.concat([grocery_df, new_row]).reset_index(drop=True)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
3       Grapes          5.00
4        Pears          4.00
'''

Setting reset_index(drop=True) resets the index to default integers. Without this step, pandas will save the original dataframes’ indices, resulting in both 'Pears' and 'Apples' sharing the same index 0.

Adding Multiple Rows to a DataFrame

For multiple rows, you can concatenate them by creating a DataFrame and adding it to the original one:

new_rows = pd.DataFrame({
    'Grocery Item': ['Avocados', 'Blueberries'],
    'Price per kg': [2.5, 10.0]
})

grocery_df = pd.concat([grocery_df, new_rows]).reset_index(drop=True)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
3       Grapes          5.00
4     Avocados          2.50
5  Blueberries         10.00
'''

You may wonder why we don’t include these rows in the original dataframe. Well, it is only sometimes possible. Imagine we have two separate grocery lists coming from different sources, for instance, from separate files. In this case, the only way to combine them into one is to use pd.concat()

Removing Rows from a DataFrame

Frequently, we must delete rows from a DataFrame. To facilitate this, Pandas provides the drop() function. Suppose you want to remove 'Grapes' or both 'Apples' and 'Oranges' from your list. Here’s how:

index_to_delete = grocery_df[grocery_df['Grocery Item'] == 'Grapes'].index

grocery_df = grocery_df.drop(index_to_delete)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
'''

Note that the .drop() method returns a new updated DataFrame instead of changing the original one. It allows you to modify the data while keeping its original state to return to it if necessary.

Removing Multiple Rows

There will be times when you will have to remove multiple rows in one go. For example, let’s say you were informed that 'Apples' and 'Oranges' are out of stock, so you need to remove them from your grocery list. The drop() function allows you to do this too.

When removing multiple rows, we utilize the .isin() function, which checks if a value exists in a particular DataFrame column. You provide it with the values you want to remove, and it outputs the indices of those rows. Let’s see it in action:

indices_to_delete = grocery_df[grocery_df['Grocery Item'].isin(['Apples', 'Oranges'])].index

grocery_df = grocery_df.drop(indices_to_delete)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
2      Bananas          2.75
3       Grapes          5.00
'''

In this block of code, the variable indices_to_delete holds the indices of the rows where the ‘Grocery Item’ is either 'Apples' or 'Oranges'. We then pass indices_to_delete to the drop() function, which removes the corresponding rows from the DataFrame.

Keep in mind, just as with removing a single row, the drop() function here doesn’t change the original DataFrame. Instead, it returns a new DataFrame with the specified rows removed. This way, you can always revert back to the original data if needed.

Recap and Practice Announcement

Congratulations! You’ve now mastered adding and removing rows in a DataFrame, a crucial element in data manipulation. We discussed rows and their indexing and learned to add rows using pd.concat() and to remove them with drop(). Now, let’s put this into practice! The upcoming exercises will enhance your data manipulation skills, enabling you to handle more complex operations on a DataFrame. Are you ready to give them a try? Well done! Now let’s experiment a bit more with our grocery list. We’re planning to make honey toast for breakfast, so we need to add ‘Honey’ to our grocery list and remove ‘Cheese’ because it’s not needed at the moment. Examine what I’ve done, then click “Run” to execute the code.

import pandas as pd

  

# Let's create a DataFrame for our grocery list

grocery_items = {'Items': ['Milk', 'Bread', 'Cheese', 'Butter'], 'Price': [2.5, 1.5, 3.5, 2.0]}

grocery_df = pd.DataFrame(grocery_items)

  

# Now let's add a new item 'Honey' to our grocery list

new_item = pd.DataFrame({'Items': ['Honey'], 'Price': [4.0]})

grocery_df = pd.concat([grocery_df, new_item]).reset_index(drop=True)

  

# But we decided we don't need 'Cheese'. Let's remove it from our list

index_to_drop = grocery_df[grocery_df['Items'] == 'Cheese'].index

grocery_df = grocery_df.drop(index_to_drop)

  

print(grocery_df)

Congratulations! You’ve now mastered adding and removing rows in a DataFrame, a crucial element in data manipulation. We discussed rows and their indexing and learned to add rows using pd.concat() and to remove them with drop(). Now, let’s put this into practice! The upcoming exercises will enhance your data manipulation skills, enabling you to handle more complex operations on a DataFrame. Are you ready to give them a try? Well done! Now let’s experiment a bit more with our grocery list. We’re planning to make honey toast for breakfast, so we need to add ‘Honey’ to our grocery list and remove ‘Cheese’ because it’s not needed at the moment. Examine what I’ve done, then click “Run” to execute the code. 做得好！现在让我们对购物清单进行更多实验。我们计划制作蜂蜜吐司作为早餐，因此我们需要将“蜂蜜”添加到我们的购物清单中并删除“奶酪”，因为目前不需要它。检查我所做的事情，然后单击“运行”来执行代码。

import pandas as pd

  

# Let's create a DataFrame for our grocery list

grocery_items = {'Items': ['Milk', 'Bread', 'Cheese', 'Butter'], 'Price': [2.5, 1.5, 3.5, 2.0]}

grocery_df = pd.DataFrame(grocery_items)

  

# Now let's add a new item 'Honey' to our grocery list

new_item = pd.DataFrame({'Items': ['Honey'], 'Price': [4.0]})

grocery_df = pd.concat([grocery_df, new_item]).reset_index(drop=True)

  

# But we decided we don't need 'Cheese'. Let's remove it from our list

index_to_drop = grocery_df[grocery_df['Items'] == 'Cheese'].index

grocery_df = grocery_df.drop(index_to_drop)

  

print(grocery_df)

输出

Items  Price  
0    Milk    2.5  
1   Bread    1.5  
3  Butter    2.0  
4   Honey    4.0

Changing the Sentence for Named Entity Recognition

Implementing Tokenization and POS Tagging

Applying Named Entity Recognition to a Sentence

Implementing a Named Entity Extraction Function

Applying NER and POS Tagging to Dataset


Feature Engineering for Text Classification

Dive deeper into the transformation of raw text data into features that machine learning models can understand. Through a practical, hands-on approach, you’ll learn everything from tokenization, generating Bag-of-Words and TF-IDF representations, to handling sparse features and applying Dimensionality Reduction techniques.

introduction and Text Data Collection

Welcome to today’s lesson! As data science and machine learning professionals, particularly in the Natural Language Processing (NLP) field, we often deal with textual data. Today, we dive into the ‘Introduction to Textual Data Collection’. Specifically, we’ll explore how to collect, understand and analyze text data using Python.

Textual data is usually unstructured, being much harder to analyze than structured data. It can take many forms, such as emails, social media posts, books, or transcripts of conversations. Understanding how to handle such data is a critical part of building effective machine learning models, especially for text classification tasks where we ‘classify’ or categorize texts. The quality of the data we use for these tasks is of utmost importance. Better, well-structured data leads to models that perform better.

The 20 Newsgroups Dataset

20 个新闻组数据集

The dataset we’ll be working with in today’s lesson is the 20 Newsgroups dataset. For some historical background, newsgroups were the precursors to modern internet forums, where people gathered to discuss specific topics. In our case, the dataset consists of approximately 20,000 documents from newsgroup discussions. These texts were originally exchanged through Usenet, a global discussion system that predates many modern Internet forums.
在今天的课程中，我们将使用的数据集是 20 个新闻组数据集。出于某种历史背景，新闻组是现代互联网论坛的前身，人们聚集在一起讨论特定主题。在我们的例子中，数据集由来自新闻组讨论的大约 20,000 个文档组成。这些文本最初是通过Usenet交换的，Usenet是一个全球讨论系统，早于许多现代互联网论坛。

The dataset is divided nearly evenly across 20 different newsgroups, each corresponding to a separate topic - this segmentation is one of the main reasons why it is especially useful for text classification tasks. The separation of data makes it excellent for training models to distinguish between different classes, or in our case, newsgroup topics.
数据集几乎平均分布在 20 个不同的新闻组中，每个新闻组对应一个单独的主题 - 这种分割是它对文本分类任务特别有用的主要原因之一。数据的分离使得训练模型能够很好地区分不同的类，或者在我们的例子中，区分新闻组主题。

From science and religion to politics and sports, the topics covered provide a diversified range of discussions. This diversity adds another layer of complexity and richness, similar to what we might experience with real-world data.
从科学和宗教到政治和体育，所涵盖的主题提供了多样化的讨论范围。这种多样性增加了另一层复杂性和丰富性，类似于我们在现实世界数据中可能遇到的情况。

Fetching and Understanding the Data Structure

获取和理解数据结构

To load this dataset, we use the fetch_20newsgroups() function from the sklearn.datasets module in Python. This function retrieves the 20 newsgroup dataset in a format that’s useful for machine learning purposes. Let’s fetch and examine the dataset.
为了加载这个数据集，我们使用 Python 中 sklearn.datasets 模块中的 fetch_20newsgroups() 函数。此函数以可用于机器学习目的的格式检索 20 个新闻组数据集。让我们获取并检查数据集。

First, let’s import the necessary libraries and fetch the data:
首先，让我们导入必要的库并获取数据：

# Importing necessary libraries
from sklearn.datasets import fetch_20newsgroups

# Fetch data
newsgroups = fetch_20newsgroups(subset='all')

「PRACTICE」

Excellent start, Space Voyager! Now, here’s a curveball. Can you fill in the missing code to create the DataFrame?

May the cosmic rays guide you!

import pandas as pd

  

# We have a list of classes and their respective teachers

classes_dict = {

'Class': ['Math', 'Science', 'English', 'Physical Education'],

'Teacher': ['Mr. James', 'Mrs. Smith', 'Ms. Williams', 'Mrs. Johnson']

}

  

# TODO: Create a DataFrame from the classes_dict

classes_df = None

  

# Display the DataFrame

print(classes_df)

解释： Excellent start, Space Voyager! Now, here’s a curveball. Can you fill in the missing code to create the DataFrame?

May the cosmic rays guide you!

import pandas as pd

  

# We have a list of classes and their respective teachers

classes_dict = {

'Class': ['Math', 'Science', 'English', 'Physical Education'],

'Teacher': ['Mr. James', 'Mrs. Smith', 'Ms. Williams', 'Mrs. Johnson']

}

  

# TODO: Create a DataFrame from the classes_dict

classes_df = None

  

# Display the DataFrame

print(classes_df)

「practice」

Time to shine, stars! Our classroom data has somewhat disappeared. Can you reconstruct it from scratch, including both the classroom data dictionary and the DataFrame? Furthermore, please add a column today, AttendancePercentage, to the DataFrame. I know you’ve got this, champ!

import pandas as pd

  

# TODO: Create a dictionary named "classroom_data" with keys: StudentName, Subject, Grade. Assign corresponding values to these keys.

  

# TODO: Create a DataFrame named "classroom_df" using the "classroom_data" dictionary. Print it.

  

# TODO: Add a new column 'AttendancePercentage' to the "classroom_df" DataFrame with values: 95, 80, 90, 85. Print the updated DataFrame.

Good try, but that’s not quite right. It seems like your code is focused on school members and teachers, but we need to modify the grocery list DataFrame instead.

Remember, we need to:

1. Add ‘Spinach’ with quantity 5.
2. Remove ‘Milk’.

Do you need help with how to start? 🤔

import pandas as pd

  

# create a DataFrame for a grocery list

data = {

'Item': ['Milk', 'Eggs', 'Bread', 'Butter'],

'Quantity': [2, 12, 1, 1]

}

  

grocery_df = pd.DataFrame(data)

  

# add a new row for 'Cheese'

new_row = pd.DataFrame({'Item': ['Cheese'], 'Quantity': [1]})

  

grocery_df = pd.concat([grocery_df, new_row]).reset_index(drop=True)

  

# remove 'Bread' from the list

item_to_remove = grocery_df[grocery_df['Item'] == 'Bread'].index

  

grocery_df = grocery_df.drop(item_to_remove)

  

print(grocery_df)

Sure, I can help with that. Here’s how you can modify your code to add ‘Spinach’ with quantity 5 and remove ‘Milk’:

import pandas as pd

# Create a DataFrame for a grocery list
data = {
    'Item': ['Milk', 'Eggs', 'Bread', 'Butter'],
    'Quantity': [2, 12, 1, 1]
}

grocery_df = pd.DataFrame(data)

# Add a new row for 'Spinach'
new_row = pd.DataFrame({'Item': ['Spinach'], 'Quantity': [5]})
grocery_df = pd.concat([grocery_df, new_row]).reset_index(drop=True)

# Remove 'Milk' from the list
grocery_df = grocery_df[grocery_df['Item'] != 'Milk']

print(grocery_df)

This code first creates a new DataFrame for ‘Spinach’ and adds it to grocery_df. It then removes the row where the ‘Item’ is ‘Milk’.

「Practice」

Fantastic work, Voyager!

However, it appears we have a small bug in our shopping list code. The item “Cheese” was supposed to be removed from the list, but it remains. Can you remedy this? Remember, it’s not necessary for you to write any code from scratch; you only need to correct one or two lines of the existing code. Good luck!

import pandas as pd

  

grocery_items = {

'Item': ['Eggs', 'Bread', 'Cheese', 'Butter'],

'Quantity': [12, 1, 1, 1]

}

  

grocery_df = pd.DataFrame(grocery_items)

  

new_items = pd.DataFrame({

'Item': ['Milk', 'Yogurt'],

'Quantity': [2, 3]

})

  

grocery_df = pd.concat([grocery_df, new_items]).reset_index(drop=True)

  

item_to_drop = grocery_df[grocery_df['Item'] == 'Cheese'].index

grocery_df.drop(item_to_drop)

  

print(grocery_df)

It looks like the drop method is not being assigned back to grocery_df. Here’s the corrected code:

import pandas as pd

grocery_items = {
    'Item': ['Eggs', 'Bread', 'Cheese', 'Butter'],
    'Quantity': [12, 1, 1, 1]
}

grocery_df = pd.DataFrame(grocery_items)

new_items = pd.DataFrame({
    'Item': ['Milk', 'Yogurt'], 
    'Quantity': [2, 3]
})

grocery_df = pd.concat([grocery_df, new_items]).reset_index(drop=True)

item_to_drop = grocery_df[grocery_df['Item'] == 'Cheese'].index
grocery_df = grocery_df.drop(item_to_drop)  # Assign the result back to grocery_df

print(grocery_df)

This code correctly removes ‘Cheese’ from the list and assigns the updated DataFrame back to grocery_df.

输出

     Item  Quantity  
0    Eggs        12  
1   Bread         1  
3  Butter         1  
4    Milk         2  
5  Yogurt         3

「Pratice]

Fantastic job, top-notch analyst!
干得好，一流的分析师！

This time around, you will be adding two new items, 'Cucumber' and 'Radish', priced at $0.50 and $0.30 respectively, to our grocery list. It’s very important to update our DataFrame once the items have been added. Ready to go the extra mile? Let’s do it!
这次，您将向我们的杂货店添加两个新商品 'Cucumber' 和 'Radish' ，价格分别为 $0.50 和 $0.30 列表。添加项目后更新我们的 DataFrame 非常重要。准备好加倍努力了吗？我们开始做吧！

import pandas as pd

  

# We will create a DataFrame for our grocery list

grocery_items = {'Items': ['Peas', 'Squash', 'Tomatoes', 'Onions'], 'Price': [1.0, 1.5, 2.5, 0.75]}

grocery_df = pd.DataFrame(grocery_items)

  

# Now let's add new items to our grocery list

  

# TODO: Create a DataFrame with two new items 'Cucumber' priced at $0.50 and 'Radish' priced at $0.30

# TODO: Concat the new items DataFrame with the current grocery DataFrame and update it

  

print(grocery_df)

Sure, here’s how you can add ‘Cucumber’ and ‘Radish’ to your grocery list and update the DataFrame:

import pandas as pd

# We will create a DataFrame for our grocery list
grocery_items = {'Items': ['Peas', 'Squash', 'Tomatoes', 'Onions'], 'Price': [1.0, 1.5, 2.5, 0.75]}
grocery_df = pd.DataFrame(grocery_items)

# Create a DataFrame with two new items 'Cucumber' priced at $0.50 and 'Radish' priced at $0.30
new_items = pd.DataFrame({
    'Items': ['Cucumber', 'Radish'],
    'Price': [0.5, 0.3]
})

# Concat the new items DataFrame with the current grocery DataFrame and update it
grocery_df = pd.concat([grocery_df, new_items]).reset_index(drop=True)

print(grocery_df)

This code creates a new DataFrame for ‘Cucumber’ and ‘Radish’ and concatenates it with the existing grocery_df. The updated DataFrame is then printed.

输出

      Items  Price  
0      Peas   1.00  
1    Squash   1.50  
2  Tomatoes   2.50  
3    Onions   0.75  
4  Cucumber   0.50  
5    Radish   0.30

「Practice」

Are you ready for your final challenge, Voyager?
航海者，你准备好迎接最后的挑战了吗？

This time, your task is to remove two specific items, 'Apples' and 'Oranges' from our grocery list. As always, do not forget to update our DataFrame post the removal of these items. Are you up for the challenge? Let’s get started!

import pandas as pd

  

# Let's begin by creating a DataFrame for our grocery list

grocery_items = {'Grocery Item': ['Peas', 'Squash', 'Tomatoes', 'Onions', 'Apples', 'Oranges'], 'Price': [1.0, 1.5, 2.5, 0.75, 0.65, 0.5]}

grocery_df = pd.DataFrame(grocery_items)

  

# Now, let's remove certain items from our grocery list

  

# TODO: Find indices of 'Apples' and 'Oranges' in the DataFrame

# TODO: Use the dataframe.drop() function to remove these items by their indices

  

print(grocery_df)

Absolutely! Here’s how you can remove ‘Apples’ and ‘Oranges’ from the grocery list and update the DataFrame:

import pandas as pd

# Let's begin by creating a DataFrame for our grocery list
grocery_items = {'Grocery Item': ['Peas', 'Squash', 'Tomatoes', 'Onions', 'Apples', 'Oranges'], 'Price': [1.0, 1.5, 2.5, 0.75, 0.65, 0.5]}
grocery_df = pd.DataFrame(grocery_items)

# Find indices of 'Apples' and 'Oranges' in the DataFrame
items_to_remove = grocery_df[grocery_df['Grocery Item'].isin(['Apples', 'Oranges'])].index

# Use the dataframe.drop() function to remove these items by their indices
grocery_df = grocery_df.drop(items_to_remove).reset_index(drop=True)

print(grocery_df)

This code finds the indices of ‘Apples’ and ‘Oranges’ in the DataFrame and then removes these items using the drop() function, updating the DataFrame accordingly.

输出

  Grocery Item  Price  
0         Peas   1.00  
1       Squash   1.50  
2     Tomatoes   2.50  
3       Onions   0.75

Lesson 6 Navigating DataFrames with Index Column and Data Locating in Pandas

Brace for liftoff! 🚀 We’re exploring the uncharted galaxies of Index Columns and Locating Elements in a Pandas DataFrame today. This is a crucial waypoint in your data analysis journey. Exciting adventures lie ahead!

Introduction and Lesson Overviews

简介和课程概述

Welcome, future data analyzers! Today, we’re tackling Index Columns and Locating Elements in a Pandas DataFrame. We’ll learn how to handle index columns, locate specific data, and strengthen our understanding of DataFrames. Ready, set, code!
欢迎，未来的数据分析者！今天，我们正在处理 Pandas DataFrame 中的索引列和定位元素。我们将学习如何处理索引列、定位特定数据并加强我们对 DataFrame 的理解。准备好，设置，编码！

Understanding the Index Column in a Pandas DataFrame

了解 Pandas DataFrame 中的索引列

In a Pandas DataFrame, an index is assigned to each row, much like the numbers on books in a library. When a DataFrame is created, Pandas establishes a default index. Let’s refer to an example:
在 Pandas DataFrame 中，为每一行分配一个索引，就像图书馆中书籍的编号一样。创建 DataFrame 时，Pandas 会建立一个默认索引。我们看一个例子：

import pandas as pd

data = {
    "Name": ["John", "Anna", "Peter", "Linda"],
    "Age": [28, 24, 35, 32],
    "City": ["New York", "Paris", "Berlin", "London"]
}

df = pd.DataFrame(data)

print(df)
"""Output:
    Name  Age      City
0   John   28  New York
1   Anna   24     Paris
2  Peter   35    Berlin
3  Linda   32    London
"""

The numbers on the left are the default index.
左边的数字是默认索引。

Setting and Modifying the Index Column

设置和修改索引列

Occasionally, we might need to establish a custom index. The Pandas’ set_index() function allows us to set a custom index. To reset the index to its default state, we use reset_index().
有时，我们可能需要建立自定义索引。 Pandas 的 set_index() 函数允许我们设置自定义索引。要将索引重置为其默认状态，我们使用 reset_index() 。

To better understand these functions, let’s consider an example in which we create an index using unique IDs:
为了更好地理解这些函数，让我们考虑一个使用唯一 ID 创建索引的示例：

df['ID'] = [101, 102, 103, 104]    # Adding unique IDs
df.set_index('ID', inplace=True)   # Setting 'ID' as index

print(df)
"""Output:
      Name  Age      City
ID                       
101   John   28  New York
102   Anna   24     Paris
103  Peter   35    Berlin
104  Linda   32    London
"""

In this example, ID column is displayed as an index. Let’s reset the index to return to the original state:
在此示例中， ID 列显示为索引。让我们重置索引以返回到原始状态：

df.reset_index(inplace=True)       # Resetting index

print(df)
"""Output:
    ID   Name  Age      City
0  101   John   28  New York
1  102   Anna   24     Paris
2  103  Peter   35    Berlin
3  104  Linda   32    London
"""

By setting inplace parameter to True, we ask pandas to reset the index in the original df dataframe. Otherwise, pandas will create a copy of the data frame with a reset index, leaving the original df untouched.
通过将 inplace 参数设置为 True ，我们要求 pandas 重置原始 df 数据框中的索引。否则，pandas 将创建具有重置索引的数据帧的副本，而原始 df 保持不变。

Locating Elements in a DataFrame

定位 DataFrame 中的元素

Let’s consider a dataframe with a custom index. If you want to select a specific row based on its index value (for example, ID = 102), you can do this:
让我们考虑一个具有自定义索引的数据框。如果您想根据索引值选择特定行（例如 ID = 102 ），您可以执行以下操作：

import pandas as pd

data = {
    "Name": ["John", "Anna", "Peter", "Linda"],
    "Age": [28, 24, 35, 32],
    "City": ["New York", "Paris", "Berlin", "London"]
}

df = pd.DataFrame(data)
df['ID'] = [101, 102, 103, 104]    # Adding unique IDs
df.set_index('ID', inplace=True)   # Setting 'ID' as index

print(df.loc[102])
'''Output:
Name     Anna
Age        24
City    Paris
Name: 102, dtype: object
'''

Selecting Multiple Rows with loc

使用“loc”选择多行

For multiple rows, simply use list of ids:
对于多行，只需使用 id 列表：

print(df.loc[[102, 104]])

'''Output:
      Name  Age    City
ID                     
102   Anna   24   Paris
104  Linda   32  London
'''

As you can see, the output of the .loc operation is some subset of the original dataframe.
如您所见， .loc 操作的输出是原始数据帧的某个子集。

Selecting Multiple Columns with loc

使用“loc”选择多列

To select specific multiple columns for these rows, you can provide the column labels as well:
要为这些行选择特定的多列，您还可以提供列标签：

print(df.loc[[102, 104], ['Name', 'Age']])
'''Output:
      Name  Age
ID             
102   Anna   24
104  Linda   32
'''

Also you can select all rows for specific columns, providing : as a set of index labels:
您还可以选择特定列的所有行，提供 : 作为一组索引标签：

print(df.loc[:, ['Name', 'Age']])
'''Output:
      Name  Age
ID             
101   John   28
102   Anna   24
103  Peter   35
104  Linda   32
'''

Using iloc for Location by Index Position

使用“iloc”按索引位置定位

The iloc function enables us to select elements in a data frame based on their index positions. iloc works like the loc, but it expects the index number of the rows. For example, we can select the 3rd row:
iloc 函数使我们能够根据索引位置选择数据框中的元素。 iloc 的工作方式与 loc 类似，但它需要行的索引号。例如，我们可以选择 3 rd 行：

print(df.iloc[3])
'''Output:
Name     Linda
Age         32
City    London
Name: 104, dtype: object
'''

You can also use slicing here:
您还可以在此处使用切片：

print(df.iloc[1:3])
'''Output:
      Name  Age    City
ID                     
102   Anna   24   Paris
103  Peter   35  Berlin
'''

Lesson Summary and Next Steps

课程总结和后续步骤

That’s it! We’ve covered the index column, how to set it, and how to locate data in a DataFrame. Exciting exercises are up next. Let’s practice and strengthen the skills you’ve learned today. Let the fun begin!
就是这样！我们已经介绍了索引列、如何设置它以及如何在 DataFrame 中定位数据。接下来是激动人心的练习。让我们练习并加强您今天学到的技能。让乐趣开始！

「practice」

Hey, remember the people dataframe? How about exploring it further? In this task, you’ll see how we can select only the 'Name' and 'Age' columns for all rows. No coding is needed here, ace, just hit the 'Run' button!
嘿，还记得 people 数据框吗？进一步探索如何？在此任务中，您将看到我们如何为所有行仅选择 'Name' 和 'Age' 列。这里不需要编码，

import pandas as pd

  

people = {

"Name": ["Austin", "Emma", "Ethan", "Sophia"],

"Age": [22, 28, 35, 30],

"City": ["Chicago", "Boston", "San Francisco", "Los Angeles"]

}

  

df = pd.DataFrame(people)

  

df['ID'] = [201, 202, 203, 204]

df.set_index('ID', inplace=True)

  

print(df.loc[:, ['Name', 'Age']])

输出

    Name  Age  
ID                
201  Austin   22  
202    Emma   28  
203   Ethan   35  
204  Sophia   30

「Practice」

Fantastic job so far! Now, we’re going to use your people database. This time, could you alter the starter code to select just the last two rows using index positions? To accomplish this, use the iloc function. Have fun!

import pandas as pd

  

people = {

"Name": ["Mike", "Emily", "Nick", "Chloe"],

"Age": [22, 28, 30, 26],

"City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]

}

  

df = pd.DataFrame(people)

  

df['ID'] = [301, 302, 303, 304]

df.set_index('ID', inplace=True)

  

# TODO: Change to select the last two rows

print(df.iloc[1:3])

Here’s the modified code to select the last two rows using the iloc function:

import pandas as pd

people = {
    "Name": ["Mike", "Emily", "Nick", "Chloe"],
    "Age": [22, 28, 30, 26],
    "City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]
}

df = pd.DataFrame(people)

df['ID'] = [301, 302, 303, 304]
df.set_index('ID', inplace=True)

# Select the last two rows using iloc
print(df.iloc[-2:])

This code correctly selects and prints the last two rows of the DataFrame.

「Practice」

Fantastic job so far! Now, we’re going to use your people database. This time, could you alter the starter code to select just the last two rows using index positions? To accomplish this, use the iloc function. Have fun!

import pandas as pd

  

people = {

"Name": ["Mike", "Emily", "Nick", "Chloe"],

"Age": [22, 28, 30, 26],

"City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]

}

  

df = pd.DataFrame(people)

  

df['ID'] = [301, 302, 303, 304]

df.set_index('ID', inplace=True)

  

# TODO: Change to select the last two rows

print(df.iloc[1:3])

解释 “To select the last two rows using the iloc function, you need to adjust the indexing appropriately. Here is the modified code:

import pandas as pd

people = {
    "Name": ["Mike", "Emily", "Nick", "Chloe"],
    "Age": [22, 28, 30, 26],
    "City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]
}

df = pd.DataFrame(people)

df['ID'] = [301, 302, 303, 304]
df.set_index('ID', inplace=True)

# Select the last two rows
print(df.iloc[-2:])

This code will correctly select and display the last two rows of the DataFrame.”

「practice」

Bravo, explorer! Here is a tricky one for you.

The same dataframe was intended to select all rows for the 'Name' and 'City' columns, but it threw an error. Could you debug it?

import pandas as pd

  

people = {

"Name": ["Lucas", "Mia", "Jack", "Sophie"],

"Age": [21, 29, 32, 25],

"City": ["Dublin", "Cork", "Galway", "Limerick"]

}

  

df = pd.DataFrame(people)

  

df['ID'] = [701, 702, 703, 704]

df.set_index('ID', inplace=True)

  

# Selecting all rows for 'Name' and 'City' columns

print(df.loc[:, [0, 2]])

输出 “The error occurs because you’re using integer-based column indexing (which iloc supports) with the loc function, which expects column labels. To select all rows for the ‘Name’ and ‘City’ columns using their labels, you should modify the code as follows:

import pandas as pd

people = {
    "Name": ["Lucas", "Mia", "Jack", "Sophie"],
    "Age": [21, 29, 32, 25],
    "City": ["Dublin", "Cork", "Galway", "Limerick"]
}

df = pd.DataFrame(people)

df['ID'] = [701, 702, 703, 704]
df.set_index('ID', inplace=True)

# Selecting all rows for 'Name' and 'City' columns
print(df.loc[:, ['Name', 'City']])

This code will correctly select and display all rows for the ‘Name’ and ‘City’ columns.”

Lessons and practices

Introduction to Textual Data Collection in NLP

NLP 中的文本数据收集简介 Introduction to Textual Data Collection in NLP
NLP 中的文本数据收集简介

Introduction and Text Data Collection

简介和文本数据收集

Welcome to today’s lesson! As data science and machine learning professionals, particularly in the Natural Language Processing (NLP) field, we often deal with textual data. Today, we dive into the ‘Introduction to Textual Data Collection’. Specifically, we’ll explore how to collect, understand and analyze text data using Python.
欢迎来到今天的课程！作为数据科学和机器学习专业人士，特别是在自然语言处理 (NLP) 领域，我们经常处理文本数据。今天，我们深入探讨“文本数据收集简介”。具体来说，我们将探索如何使用 Python 收集、理解和分析文本数据。

Textual data is usually unstructured, being much harder to analyze than structured data. It can take many forms, such as emails, social media posts, books, or transcripts of conversations. Understanding how to handle such data is a critical part of building effective machine learning models, especially for text classification tasks where we ‘classify’ or categorize texts. The quality of the data we use for these tasks is of utmost importance. Better, well-structured data leads to models that perform better.
文本数据通常是非结构化的，比结构化数据更难分析。它可以采取多种形式，例如电子邮件、社交媒体帖子、书籍或对话记录。了解如何处理此类数据是构建有效的机器学习模型的关键部分，特别是对于我们对文本进行“分类”或分类的文本分类任务。我们用于这些任务的数据的质量至关重要。更好、结构良好的数据可以带来性能更好的模型。

The 20 Newsgroups Dataset

20 个新闻组数据集

The dataset we’ll be working with in today’s lesson is the 20 Newsgroups dataset. For some historical background, newsgroups were the precursors to modern internet forums, where people gathered to discuss specific topics. In our case, the dataset consists of approximately 20,000 documents from newsgroup discussions. These texts were originally exchanged through Usenet, a global discussion system that predates many modern Internet forums.
我们在今天的课程中将使用的数据集是 20 个新闻组数据集。从某些历史背景来看，新闻组是现代互联网论坛的先驱，人们聚集在一起讨论特定主题。在我们的例子中，数据集包含来自新闻组讨论的大约 20,000 个文档。这些文本最初是通过 Usenet 进行交换的，这是一个早于许多现代互联网论坛的全球讨论系统。

The dataset is divided nearly evenly across 20 different newsgroups, each corresponding to a separate topic - this segmentation is one of the main reasons why it is especially useful for text classification tasks. The separation of data makes it excellent for training models to distinguish between different classes, or in our case, newsgroup topics.
该数据集几乎均匀地分布在 20 个不同的新闻组中，每个新闻组对应一个单独的主题 - 这种分割是它对于文本分类任务特别有用的主要原因之一。数据的分离使得训练模型能够很好地区分不同的类别，或者在我们的例子中区分新闻组主题。

From science and religion to politics and sports, the topics covered provide a diversified range of discussions. This diversity adds another layer of complexity and richness, similar to what we might experience with real-world data.
从科学和宗教到政治和体育，所涵盖的主题提供了多元化的讨论。这种多样性增加了另一层复杂性和丰富性，类似于我们在现实世界数据中可能遇到的情况。

# Understanding the structure of the data
print("\n\nData Structure\n-------------")
print(f'Type of data: {type(newsgroups.data)}')
print(f'Type of target: {type(newsgroups.target)}')

We are fetching the data and observing the type of the data and target. The type of data tells us what kind of data structure is used to store the text data while the type of target shouts what type of structure is used to store the labels. Here is what the output looks like:
我们正在获取数据并观察 data 和 target 的类型。 type of data 告诉我们使用什么类型的数据结构来存储文本数据，而 type of target 告诉我们使用什么类型的结构来存储标签。输出如下所示：

Data Structure
-------------
Type of data: <class 'list'>
Type of target: <class 'numpy.ndarray'>

As printed out, the data is stored as a list, and target as a numpy array.
打印出来时， data 存储为列表， target 存储为 numpy 数组。

Diving Into Data Exploration

深入数据探索

Now, let’s explore the data points, target variables and the potential classes in the dataset:
现在，让我们探索数据集中的数据点、目标变量和潜在类别：

print("\n\nData Exploration\n----------------")
print(f'Number of datapoints: {len(newsgroups.data)}')
print(f'Number of target variables: {len(newsgroups.target)}')
print(f'Possible classes: {newsgroups.target_names}')

We get the length of the data list to fetch the number of data points. Also, we get the length of the target array. Lastly, we fetch the possible classes or newsgroups in the dataset. Here is what we get:
我们获取 data 列表的长度来获取数据点的数量。此外，我们还获得了 target 数组的长度。最后，我们获取数据集中可能的类或新闻组。这是我们得到的：

Data Exploration
----------------
Number of datapoints: 18846
Number of target variables: 18846
Possible classes: ['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc']

print("\n\nSample datapoint\n----------------")
print(f'\nArticle:\n-------\n{newsgroups.data[10]}')
print(f'\nCorresponding Topic:\n------------------\n{newsgroups.target_names[newsgroups.target[10]]}')

The Article fetched is the 10th article in the dataset and Corresponding Topic is the actual topic that the article belongs to. Here’s the output:
获取的 Article 是数据集中的第 10 篇文章， Corresponding Topic 是该文章所属的实际主题。这是输出：

Sample datapoint
----------------

Article:
-------
From: sandvik@newton.apple.com (Kent Sandvik)
Subject: Re: 14 Apr 93   God's Promise in 1 John 1: 7
Organization: Cookamunga Tourist Bureau
Lines: 17

In article <1qknu0INNbhv@shelley.u.washington.edu>, > Christian:  washed in
the blood of the lamb.
> Mithraist:  washed in the blood of the bull.
> 
> If anyone in .netland is in the process of devising a new religion,
> do not use the lamb or the bull, because they have already been
> reserved.  Please choose another animal, preferably one not
> on the Endangered Species List.  

This will be a hard task, because most cultures used most animals
for blood sacrifices. It has to be something related to our current
post-modernism state. Hmm, what about used computers?

Cheers,
Kent
---
sandvik@newton.apple.com. ALink: KSAND -- Private activities on the net.


Corresponding Topic:
------------------
talk.religion.misc

Lesson Summary 课程总结

Nice work! Through today’s lesson, you’ve learned to fetch and analyze text data for text classification. If you’ve followed along, you should now understand the structure of text data and how to fetch and analyze it using Python.
干得好！通过今天的课程，您已经学会了如何获取和分析文本数据以进行文本分类。如果您已经跟进，现在应该了解文本数据的结构以及如何使用 Python 获取和分析它。

But our journey to text classification is just starting. In upcoming lessons, we’ll dive deeper into related topics such as cleaning textual data, handling missing values, and restructuring textual data for analysis. Each step forward improves your expertise in text classification. Keep going!
但我们的文本分类之旅才刚刚开始。在接下来的课程中，我们将更深入地探讨相关主题，例如清理文本数据、处理缺失值以及重构文本数据以进行分析。每前进一步都会提高您在文本分类方面的专业知识。继续前进！

Sample Data Preview 样本数据预览

Lastly, let’s fetch and understand what a sample data point and its corresponding label looks like:
最后，让我们获取并了解示例数据点及其相应标签的样子：

Fetching and Understanding the Data Structure

获取和理解数据结构

To load this dataset, we use the fetch_20newsgroups() function from the sklearn.datasets module in Python. This function retrieves the 20 newsgroup dataset in a format that’s useful for machine learning purposes. Let’s fetch and examine the dataset.
为了加载此数据集，我们使用 Python 中 sklearn.datasets 模块中的 fetch_20newsgroups() 函数。此函数以对机器学习有用的格式检索 20 个新闻组数据集。让我们获取并检查数据集。

First, let’s import the necessary libraries and fetch the data:
首先，让我们导入必要的库并获取数据：

Python

CopyPlay

1# Importing necessary libraries 2from sklearn.datasets import fetch_20newsgroups 3 4# Fetch data 5newsgroups = fetch_20newsgroups(subset='all')

The datasets fetched from sklearn typically have three attributes—data, target, and target_names. data refers to the actual content, target refers to the labels for the texts, and target_names provides names for the target labels.
从 sklearn 获取的数据集通常具有三个属性 - data 、 target 和 target_names 。 data 指实际内容， target 指文本标签， target_names 提供目标标签的名称。

Next, let’s understand the structure of the fetched data:
接下来我们来了解一下获取到的数据的结构：

「Practice1」Explore More of the 20 Newsgroups Dataset

探索 20 个新闻组数据集的更多内容 Excellent job, Space Voyager! Now, make a small alteration to the starter code: change it to print out the first 150 characters of the 500th article from our 20 Newsgroups dataset, and also display its corresponding topic.

# Import necessary libraries and modules

from sklearn.datasets import fetch_20newsgroups

  

# Fetch the dataset

newsgroups = fetch_20newsgroups(subset='all')

  

# Here we are fetching the first 100 characters of the 200th article and its corresponding topic

print(f'\nArticle:')

print(f'{newsgroups.data[200][:100]}')

print(f'\nCorresponding Topic:')

print(f'{newsgroups.target_names[newsgroups.target[200]]}')

“Here is the updated code to print out the first 150 characters of the 500th article from the 20 Newsgroups dataset and display its corresponding topic.”

# Import necessary libraries and modules
from sklearn.datasets import fetch_20newsgroups

# Fetch the dataset
newsgroups = fetch_20newsgroups(subset='all')

# Fetch the first 150 characters of the 500th article and its corresponding topic
print(f'\nArticle:')
print(f'{newsgroups.data[499][:150]}')
print(f'\nCorresponding Topic:')
print(f'{newsgroups.target_names[newsgroups.target[499]]}')

输出

Article:  
From: ry01@ns1.cc.lehigh.edu (ROBERT YUNG)  
Subject: How long do monitors last????  
Article-I.D.: ns1.1993Apr5.200422.65952  
Organization: Lehigh Univers  
  
Corresponding Topic:  
comp.sys.ibm.pc.hardware

「Practice2」Uncover the End of 20 Newsgroups Dataset

Celestial Traveler, your journey continues! Fill in the blanks (____) to import and explore our dataset. We aim to extract and display the last three articles and their corresponding topics. Can you reveal what’s at the end of our dataset?

# Import necessary libraries

from sklearn.datasets import fetch_20newsgroups

  

# Fetch the dataset

newsgroups = ____(subset='all')

  

# Fetch last three articles

last_three_articles = newsgroups.____[-3:]

  

# Fetch the corresponding topics

corresponding_topics = [newsgroups.target_names[i] for i in newsgroups.____[-3:]]

  

# Display Last three articles and their corresponding topics

for i in range(3):

print(f'\nLast article {i+1}:')

print(f'{last_three_articles[i]}\n')

print(f'Corresponding Topic {i+1}:')

print(f'{corresponding_topics[i]}\n')

“Here is the completed code to import and explore the dataset, extracting and displaying the last three articles and their corresponding topics.”

# Import necessary libraries
from sklearn.datasets import fetch_20newsgroups

# Fetch the dataset
newsgroups = fetch_20newsgroups(subset='all')

# Fetch last three articles
last_three_articles = newsgroups.data[-3:]

# Fetch the corresponding topics
corresponding_topics = [newsgroups.target_names[i] for i in newsgroups.target[-3:]]

# Display last three articles and their corresponding topics
for i in range(3):
    print(f'\nLast article {i+1}:')
    print(f'{last_three_articles[i]}\n')
    print(f'Corresponding Topic {i+1}:')
    print(f'{corresponding_topics[i]}\n')

输出

  
Last article 1:  
From: westes@netcom.com (Will Estes)  
Subject: Mounting CPU Cooler in vertical case  
Organization: Mail Group  
X-Newsreader: TIN [version 1.1 PL8]  
Lines: 13  
  
I just installed a DX2-66 CPU in a clone motherboard, and tried mounting a CPU   
cooler on the chip.  After about 1/2 hour, the weight of the cooler was enough   
to dislodge the CPU from its mount.  It ended up bending a few pins  
on the CPU, but luckily the power was not on yet.  I ended up  
pressing the CPU deeply into its socket and then putting the CPU  
cooler back on.  So far so good.  
  
Have others had this problem?  How do you ensure that the weight of  
the CPU fan and heatsink do not eventually work the CPU out of its  
socket when mounting the motherboard in a vertical case?  
  
--   
Will Estes		Internet: westes@netcom.com  
  
  
Corresponding Topic 1:  
comp.sys.ibm.pc.hardware  
  
  
Last article 2:  
From: steve@hcrlgw (Steven Collins)  
Subject: Re: Sphere from 4 points?  
Organization: Central Research Lab. Hitachi, Ltd.  
Lines: 27  
Nntp-Posting-Host: hcrlgw  
  
In article <1qkgbuINNs9n@shelley.u.washington.edu> bolson@carson.u.washington.edu (Edward Bolson) writes:  
>Boy, this will be embarassing if it is trivial or an FAQ:  
>  
>Given 4 points (non coplanar), how does one find the sphere, that is,  
>center and radius, exactly fitting those points?  I know how to do it  
>for a circle (from 3 points), but do not immediately see a   
>straightforward way to do it in 3-D.  I have checked some  
>geometry books, Graphics Gems, and Farin, but am still at a loss?  
>Please have mercy on me and provide the solution?    
  
Wouldn't this require a hyper-sphere.  In 3-space, 4 points over specifies  
a sphere as far as I can see.  Unless that is you can prove that a point  
exists in 3-space that is equi-distant from the 4 points, and this may not  
necessarily happen.  
  
Correct me if I'm wrong (which I quite possibly am!)  
  
steve  
---  
  
  
  
--   
+---------------------------------------+--------------------------------+  
| Steven Collins			| email: steve@crl.hitachi.co.jp |  
| Visiting Computer Graphics Researcher	| phone: (0423)-23-1111 	 |  
| Hitachi Central Research Lab. Tokyo.	| fax:   (0423)-27-7742		 |  
  
  
Corresponding Topic 2:  
comp.graphics  
  
  
Last article 3:  
From: chriss@netcom.com (Chris Silvester)  
Subject: "Production Hold" on '93 Firebird/Camaro w/ 6-Speed  
Organization: Netcom - Online Communication Services (408 241-9760 guest)  
Distribution: usa  
Lines: 30  
  
After a tip from Gary Crum (crum@fcom.cc.utah.edu) I got on the Phone  
with "Pontiac Systems" or "Pontaic Customer Service" or whatever, and  
inquired about a rumoured Production Hold on the Formula Firebird and  
Trans Am.  BTW, Talking with the dealer I bought the car from got me  
nowhere.  After being routed to a "Firebird Specialist", I was able  
to confirm that this is in fact the case.  
  
At first, there was some problem with the 3:23 performance axle ratio.  
She wouldn't go into any details, so I don't know if there were some  
shipped that had problems, or if production was held up because they  
simply didn't have the proper parts from the supplier.  As I say, she  
was pretty vague on that, so if anyone else knows anything about this,  
feel free to respond.  Supposedly, this problem is now solved.  
  
Second, there is a definate shortage of parts that is somehow related  
to the six-speed Manual transmission.  So as of this posting, there is  
a production hold on these cars.  She claimed part of the delay was  
not wanting to use inferior quality parts for the car, and therefore  
having to wait for the right high quality parts...  I'm not positive  
that this applies to the Camaro as well, but I'm guessing it would.  
  
Can anyone else shed some light on this?  
  
Chris S.  
--   
--------------------------------------------------------------------------------  
Chris Silvester      | "Any man capable of getting himself elected President  
chriss@sam.amgen.com |  should by no means be allowed to do the job"  
chriss@netcom.com    |   - Douglas Adams, The Hitchhiker's Guide to the Galaxy  
--------------------------------------------------------------------------------  
  
  
Corresponding Topic 3:  
rec.autos

「Practice 3」 Fetch Specific Categories from Dataset

从数据集中获取特定类别 Celestial Traveler, let’s narrow down our data collection. Modify the provided code to fetch only the 'alt.atheism' and 'talk.religion.misc' categories from our dataset. Then, display the first two articles from these categories along with their corresponding labels.

# Import necessary libraries

from sklearn.datasets import fetch_20newsgroups

  

# Fetch a subset of the dataset containing selected categories. Update the categories as needed.

newsgroups_subset = fetch_20newsgroups(subset='all', categories=['comp.graphics', 'sci.space'])

  

# Display the first two articles and their corresponding topics from this subset

for i in range(2):

print(f'\nArticle {i+1}:')

print(f'{newsgroups_subset.data[i]}\n')

print(f'Corresponding Topic {i+1}:')

print(f'{newsgroups_subset.target_names[newsgroups_subset.target[i]]}\n')

解题： “Here is the modified code to fetch only the 'alt.atheism' and 'talk.religion.misc' categories from the dataset and display the first two articles along with their corresponding labels.”

# Import necessary libraries
from sklearn.datasets import fetch_20newsgroups

# Fetch a subset of the dataset containing selected categories
newsgroups_subset = fetch_20newsgroups(subset='all', categories=['alt.atheism', 'talk.religion.misc'])

# Display the first two articles and their corresponding topics from this subset
for i in range(2):
    print(f'\nArticle {i+1}:')
    print(f'{newsgroups_subset.data[i]}\n')
    print(f'Corresponding Topic {i+1}:')
    print(f'{newsgroups_subset.target_names[newsgroups_subset.target[i]]}\n')

输出

  
Article 1:  
From: agr00@ccc.amdahl.com (Anthony G Rose)  
Subject: Re: Who's next?  Mormons and Jews?  
Reply-To: agr00@JUTS.ccc.amdahl.com (Anthony G Rose)  
Organization: Amdahl Corporation, Sunnyvale CA  
Lines: 18  
  
In article <1993Apr20.142356.456@ra.royalroads.ca> mlee@post.RoyalRoads.ca (Malcolm Lee) writes:  
>  
>In article <C5rLps.Fr5@world.std.com>, jhallen@world.std.com (Joseph H Allen) writes:  
>|> In article <1qvk8sINN9vo@clem.handheld.com> jmd@cube.handheld.com (Jim De Arras) writes:  
>|>   
>|> It was interesting to watch the 700 club today.  Pat Robertson said that the  
>|> "Branch Dividians had met the firey end for worshipping their false god." He  
>|> also said that this was a terrible tragedy and that the FBI really blew it.  
>  
>I don't necessarily agree with Pat Robertson.  Every one will be placed before  
>the judgement seat eventually and judged on what we have done or failed to do  
>on this earth.  God allows people to choose who and what they want to worship.  
  
I'm sorry, but He does not!  Ever read the FIRST commandment?  
  
>Worship of money is one of the greatest religions in this country.  
  
You mean, false religion!  
  
  
Corresponding Topic 1:  
talk.religion.misc  
  
  
Article 2:  
From: frank@D012S658.uucp (Frank O'Dwyer)  
Subject: Re: Tieing Abortion to Health Reform -- Is Clinton Nuts?  
Organization: Siemens-Nixdorf AG  
Lines: 21  
NNTP-Posting-Host: d012s658.ap.mchp.sni.de  
  
In article <1993Apr26.163627.11364@csrd.uiuc.edu> g-skinner@uiuc.edu writes:  
#I find myself unable to put these two statements together in a  
#sensible way:  
#  
#>Abortion is done because the mother can not afford the *pregnancy*.  
#  
#[...]  
#  
#>If we refused to pay for the more expensive choice of birth, *then*  
#>your statement would make sense.  But that is not the case, so it doesn't.  
#  
#Are we paying for the birth or not, Mr. Parker?  If so, why can't the  
#mother afford the pregnancy?  If not, what is the meaning of the  
#latter objection?  You can't have it both ways.  
  
Birth != pregnancy.  If they were the same, the topic of abortion would   
hardly arise, would it, Mr. Skinner?  
  
--   
Frank O'Dwyer                                  'I'm not hatching That'  
odwyer@sse.ie                                  from "Hens",  by Evelyn Conlon  
  
  
Corresponding Topic 2:  
talk.religion.misc

「Practice 4」Fetching the Third Article from Dataset

从数据集中获取第三篇文章 Well done, Stellar Navigator! Next, fill in the missing line in the code below to fetch and display the third article from the 20 Newsgroups dataset with its corresponding topic. Prepare your spacecraft for another adventure in data exploration!
干得好，恒星导航员！接下来，填写下面代码中缺少的行，以从 20 个新闻组数据集中获取并显示第三篇文章及其相应的主题。准备好您的航天器，进行另一次数据探索冒险！

# Import necessary libraries

from sklearn.datasets import fetch_20newsgroups

  

# Fetch the dataset

newsgroups = fetch_20newsgroups(subset='all')

  

# TODO: Fetch the third article and its corresponding topic

输出

Topic: talk.politics.mideast  
Article: From: hilmi-er@dsv.su.se (Hilmi Eren)  
Subject: Re: ARMENIA SAYS IT COULD SHOOT DOWN TURKISH PLANES (Henrik)  
Lines: 95  
Nntp-Posting-Host: viktoria.dsv.su.se  
Reply-To: hilmi-er@dsv.su.se (Hilmi Eren)  
Organization: Dept. of Computer and Systems Sciences, Stockholm University  
  
  
  
  
|>The student of "regional killings" alias Davidian (not the Davidian religios sect) writes:  
  
  
|>Greater Armenia would stretch from Karabakh, to the Black Sea, to the  
|>Mediterranean, so if you use the term "Greater Armenia" use it with care.  
  
  
	Finally you said what you dream about. Mediterranean???? That was new....  
	The area will be "greater" after some years, like your "holocaust" numbers......  
  
  
  
  
|>It has always been up to the Azeris to end their announced winning of Karabakh   
|>by removing the Armenians! When the president of Azerbaijan, Elchibey, came to   
|>power last year, he announced he would be be "swimming in Lake Sevan [in   
|>Armeniaxn] by July".  
		*****  
	Is't July in USA now????? Here in Sweden it's April and still cold.  
	Or have you changed your calendar???  
  
  
|>Well, he was wrong! If Elchibey is going to shell the   
|>Armenians of Karabakh from Aghdam, his people will pay the price! If Elchibey   
						    ****************  
|>is going to shell Karabakh from Fizuli his people will pay the price! If   
						    ******************  
|>Elchibey thinks he can get away with bombing Armenia from the hills of   
|>Kelbajar, his people will pay the price.   
			    ***************  
  
  
	NOTHING OF THE MENTIONED IS TRUE, BUT LET SAY IT's TRUE.  
	  
	SHALL THE AZERI WOMEN AND CHILDREN GOING TO PAY THE PRICE WITH  
						    **************  
	BEING RAPED, KILLED AND TORTURED BY THE ARMENIANS??????????  
	  
	HAVE YOU HEARDED SOMETHING CALLED: "GENEVA CONVENTION"???????  
	YOU FACIST!!!!!  
  
  
  
	Ohhh i forgot, this is how Armenians fight, nobody has forgot  
	you killings, rapings and torture against the Kurds and Turks once  
	upon a time!  
        
         
  
|>And anyway, this "60   
|>Kurd refugee" story, as have other stories, are simple fabrications sourced in   
|>Baku, modified in Ankara. Other examples of this are Armenia has no border   
|>with Iran, and the ridiculous story of the "intercepting" of Armenian military   
|>conversations as appeared in the New York Times supposedly translated by   
|>somebody unknown, from Armenian into Azeri Turkish, submitted by an unnamed   
|>"special correspondent" to the NY Times from Baku. Real accurate!  
  
Ohhhh so swedish RedCross workers do lie they too? What ever you say  
"regional killer", if you don't like the person then shoot him that's your policy.....l  
  
  
|>[HE]	Search Turkish planes? You don't know what you are talking about.<-------  
|>[HE]	since it's content is announced to be weapons? 				i	   
										i  
|>Well, big mouth Ozal said military weapons are being provided to Azerbaijan	i  
|>from Turkey, yet Demirel and others say no. No wonder you are so confused!	i  
										i  
										i  
	Confused?????								i  
	You facist when you delete text don't change it, i wrote:		i  
										i  
        Search Turkish planes? You don't know what you are talking about.	i  
        Turkey's government has announced that it's giving weapons  <-----------i  
        to Azerbadjan since Armenia started to attack Azerbadjan		  
        it self, not the Karabag province. So why search a plane for weapons	  
        since it's content is announced to be weapons?     
  
	If there is one that's confused then that's you! We have the right (and we do)  
	to give weapons to the Azeris, since Armenians started the fight in Azerbadjan!  
   
  
|>You are correct, all Turkish planes should be simply shot down! Nice, slow  
|>moving air transports!  
  
	Shoot down with what? Armenian bread and butter? Or the arms and personel   
	of the Russian army?  
  
  
  
  
Hilmi Eren  
Stockholm University

Exploring Text Length in Newsgroups Dataset

探索新闻组数据集中的文本长度 Great job, Space Voyager! Now, as a final task, write a Python script that calculates and displays the lengths of the first five articles (in terms of the number of characters) from the 20 Newsgroups dataset.
干得好，太空航行者！现在，作为最后一项任务，编写一个 Python 脚本，计算并显示 20 个新闻组数据集中的前 5 篇文章的长度（以字符数计）。

# Import necessary libraries and modules

from sklearn.datasets import fetch_20newsgroups

  

# TODO: Fetch the 20 Newsgroups dataset

  

# TODO: Iterate over the first five articles,

# TODO: Calculate their length in terms of the number of characters and display it

解释

 Import necessary libraries and modules
from sklearn.datasets import fetch_20newsgroups

# Fetch the 20 Newsgroups dataset
newsgroups = fetch_20newsgroups(subset='all')

# Iterate over the first five articles,
# Calculate their length in terms of the number of characters and display it
for i in range(5):
    article_length = len(newsgroups.data[i])
    print(f"Length of article {i+1}: {article_length} characters")

输出

Length of article 1: 902 characters  
Length of article 2: 963 characters  
Length of article 3: 3780 characters  
Length of article 4: 3096 characters  
Length of article 5: 910 characters

Lesson 2 第2课

Mastering Text Cleaning for NLP: Techniques and Applications
掌握 NLP 文本清理：技术与应用

Lesson 2

Mastering Text Cleaning for NLP: Techniques and Applications

Introduction 介绍

Welcome to today’s lesson on Text Cleaning Techniques! In any Natural Language Processing (NLP) project, the quality of your results depends heavily on the quality of your input. Hence, cleaning our textual data becomes critical for the accuracy of our project. Our main objective for today is to delve into how to clean textual data using Python. By the end of this session, you will be comfortable with creating and applying a simple text cleaning pipeline in Python.
欢迎来到今天的文本清理技术课程！在任何自然语言处理 (NLP) 项目中，结果的质量在很大程度上取决于输入的质量。因此，清理文本数据对于我们项目的准确性至关重要。我们今天的主要目标是深入研究如何使用 Python 清理文本数据。在本课程结束时，您将能够轻松地在 Python 中创建和应用简单的文本清理管道。

Introduction

Welcome to today’s lesson on Text Cleaning Techniques! In any Natural Language Processing (NLP) project, the quality of your results depends heavily on the quality of your input. Hence, cleaning our textual data becomes critical for the accuracy of our project. Our main objective for today is to delve into how to clean textual data using Python. By the end of this session, you will be comfortable with creating and applying a simple text cleaning pipeline in Python.

Understanding Text Cleaning

Text cleaning is essential in NLP, involving the preparation of text data for analysis. Why is it necessary? Imagine trying to perform text classification on social media posts; people often use colloquial language, abbreviations, and emojis. In many cases, posts might also be in different languages. These variations make it challenging for machines to understand context without undergoing preprocessing.

We get rid of superfluous variations and distractions to make the text understandable for algorithms, thereby increasing accuracy. These distractions could range from punctuation, special symbols, numbers, to even common words that do not carry significant meaning (commonly referred to as “stop words”).

Python’s Regex (Regular Expression) library, re, is an ideal tool for such text cleaning tasks, as it is specifically designed to work with string patterns. Within this library, we will be using re.sub, a method employed to replace parts of a string. This method operates by accepting three arguments: re.sub(pattern, repl, string). Here, pattern is the character pattern we’re looking to replace, repl is the replacement string, and string is the text being processed. In essence, any part of the string argument that matches the pattern argument gets replaced by the repl argument.

As we proceed, a clearer understanding of the functionality and application of re.sub will be provided. Now, let’s delve into it!

Text Cleaning Process

The text cleaning process comprises multiple steps where each step aims to reduce the complexity of the text. Let’s take you through the process using a Python function, clean_text.

import re

def clean_text(text):
    text = text.lower()  # Convert text to lower case
    text = re.sub(r'\S*@\S*\s?', '', text)  # Remove email addresses
    text = re.sub(r'http\S+', '', text)  # Remove URLs
    text = re.sub(r'\W', ' ', text)  # Remove punctuation and special characters
    text = re.sub(r'\d', ' ', text)  # Remove digits
    text = re.sub(r'\s\s+', ' ', text)  # Remove extra spaces

    return text

In the function above we can see how each line corresponds to a step in the cleaning process:

1. Lowercase: We convert all text to lower case, so every word looks the same unless it carries a different meaning. This way, words like ‘The’ and ’the’ are no longer seen as different.
2. Email addresses: Email addresses don’t usually provide useful information unless we’re specifically looking for them. This line of code removes any email addresses found.
3. URLs: Similarly, URLs are removed as they are typically not useful in text classification tasks.
4. Special Characters: We remove any non-word characters (\W) and replace it with space using regex. This includes special characters and punctuation.
5. Numbers: We’re dealing with text data, so numbers are also considered distractions unless they carry significant meaning.
6. Extra spaces: Any resulting extra spaces from the previous steps are removed.

Let’s go ahead and run this function on some demo input to see it in action!

print(clean_text('Check out the course at www.codesignal.com/course123'))

The output of the above code will be:

check out the course at www codesignal com course 

Implementing Text Cleaning Function

Now that you are familiar with the workings of the function let’s implement it in the 20 Newsgroups dataset.

To apply our cleaning function on the dataset, we will make use of the DataFrame data structure from Pandas, another powerful data manipulation tool in Python.

import pandas as pd
from sklearn.datasets import fetch_20newsgroups

# Fetching the 20 Newsgroups Dataset
newsgroups_data = fetch_20newsgroups(subset='train')
nlp_df = pd.DataFrame(newsgroups_data.data, columns = ['text'])

# Applied the cleaning function to the text data
nlp_df['text'] = nlp_df['text'].apply(lambda x: clean_text(x))

# Checking the cleaned text
print(nlp_df.head())

The output of the above code will be:

                                                text
0  from where s my thing subject what car is this...
1  from guy kuo subject si clock poll final call ...
2  from thomas e willis subject pb questions orga...
3  from joe green subject re weitek p organizatio...
4  from jonathan mcdowell subject re shuttle laun...

In this code, we’re applying the clean_text function to each ’text’ in our DataFrame using the apply function. The apply function passes every value of the DataFrame column to the clean_text function one by one.

Understanding Effectiveness of Text Cleaning Function

We want to understand the impact of our text cleaning function. We can achieve this by looking at our text before and after cleaning. Let’s use some new examples:

test_texts = ['This is an EXAMPLE!', 'Another ex:ample123 with special characters $#@!', 'example@mail.com is an email address.']
for text in test_texts:
    print(f'Original: {text}')
    print(f'Cleaned: {clean_text(text)}')
    print('--')

The output of the above code will be:

Original: This is an EXAMPLE!
Cleaned: this is an example 
--
Original: Another ex:ample123 with special characters $#@!
Cleaned: another ex ample with special characters 
--
Original: example@mail.com is an email address.
Cleaned: is an email address 
--

In the example above, you will see that our function successfully transforms all text to lower case and removes punctuation, digits, and email addresses!

Lesson Summary and Practice Exercises

Today we delved into the text cleaning process in Natural Language Processing. We shared why it is necessary and how to implement it in Python. We then applied our text cleaning function on a textual dataset.

We have a few exercises lined up based on what we learned today. Keep swimming ahead, and remember, you learn the most by doing. Happy cleaning!

「practice」

Well done, Space Voyager! Now, to further explore the workings of our text cleaning function, let’s use a different sentence. Replace the first sentence in the test_texts list with the phrase “I love learning at CodeSignal; it’s so interactive and fun!”. Then run the clean_text function with the updated list.

import re

  

def clean_text(text):

text = text.lower() # Convert text to lower case

text = re.sub(r'\S*@\S*\s?', '', text) # Remove email addresses

text = re.sub(r'http\S+', '', text) # Remove URLs

text = re.sub(r'[^\w\s]', '', text) # Remove punctuation and special characters but leave spaces

text = re.sub(r'\d+', '', text) # Remove digits

text = re.sub(r'\s+', ' ', text) # Normalize whitespace to single spaces

  

return text.strip() # Strip leading and trailing spaces

  

test_texts = ['This is an EXAMPLE!',

'Another ex:ample123 with special characters $#@!',

'example@mail.com is an email address.']

for text in test_texts:

print(f'Original: {text}')

print(f'Cleaned: {clean_text(text)}')

print('--')

Certainly! Here is the updated code with the new sentence in the test_texts list and the clean_text function:

import re

def clean_text(text):
    text = text.lower()  # Convert text to lower case
    text = re.sub(r'\S*@\S*\s?', '', text)  # Remove email addresses
    text = re.sub(r'http\S+', '', text)  # Remove URLs
    text = re.sub(r'[^\w\s]', '', text)  # Remove punctuation and special characters but leave spaces
    text = re.sub(r'\d+', '', text)  # Remove digits
    text = re.sub(r'\s+', ' ', text)  # Normalize whitespace to single spaces
    return text.strip()  # Strip leading and trailing spaces

test_texts = ["I love learning at CodeSignal; it's so interactive and fun!",
              'Another ex:ample123 with special characters $#@!',
              'example@mail.com is an email address.']

for text in test_texts:
    print(f'Original: {text}')
    print(f'Cleaned: {clean_text(text)}')
    print('--')

Running this updated code will process the new sentence along with the other test cases. Here is the expected output:

Original: I love learning at CodeSignal; it's so interactive and fun!
Cleaned: i love learning at codesignal it s so interactive and fun
--
Original: Another ex:ample123 with special characters $#@!
Cleaned: another example with special characters 
--
Original: example@mail.com is an email address.
Cleaned: is an email address
--

This demonstrates how the clean_text function processes and cleans the text by converting to lowercase, removing punctuation, digits, extra spaces, and other unwanted characters.

「Practice」

Superb job! Now, let’s ensure you have grasped the core Python functions and regex patterns used in text cleaning.

Fill in the blanks (____) to transform all text to lowercase and remove extra spaces.

import re

  

def clean_text(text):

text = text.____() # Convert text to lower case

text = re.sub(r'\S*@\S*\s?', '', text) # Remove email addresses

text = re.sub(r'http\S+', '', text) # Remove URLs

text = re.sub(r'\W', ' ', text) # Remove punctuation and special characters

text = re.sub(r'\d', ' ', text) # Remove digits

text = re.sub(r'\____\s+', ' ', text) # Remove extra spaces

return text.strip()

  

test_texts = ['Fill me In! I enjoy problem-solving tasks like these; they are quite engaging.',

'Another ex:ample123 with special characters $#@!',

'example@mail.com is an email address.']

for text in test_texts:

print(f'Original: {text}')

print(f'Cleaned: {clean_text(text)}')

print('--')

Here is the Python code with the blanks filled in to transform all text to lowercase and remove extra spaces:

import re

def clean_text(text):
    text = text.lower() # Convert text to lower case
    text = re.sub(r'\S*@\S*\s?', '', text) # Remove email addresses
    text = re.sub(r'http\S+', '', text) # Remove URLs
    text = re.sub(r'\W', ' ', text) # Remove punctuation and special characters
    text = re.sub(r'\d', ' ', text) # Remove digits
    text = re.sub(r'\s+', ' ', text) # Remove extra spaces
    
    return text.strip()

test_texts = [
    'Fill me In! I enjoy problem-solving tasks like these; they are quite engaging.',
    'Another ex:ample123 with special characters $#@!',
    'example@mail.com is an email address.'
]

for text in test_texts:
    print(f'Original: {text}')
    print(f'Cleaned: {clean_text(text)}')
    print('--')

Explanation of the substitutions:

• text.lower(): Converts the text to lowercase.
• re.sub(r'\S*@\S*\s?', '', text): Removes email addresses by matching patterns of the form ‘something@something’.
• re.sub(r'http\S+', '', text): Removes URLs by matching patterns starting with ‘http’.
• re.sub(r'\W', ' ', text): Removes punctuation and special characters by replacing non-word characters with a space.
• re.sub(r'\d', ' ', text): Removes digits by replacing them with a space.
• re.sub(r'\s+', ' ', text): Replaces multiple spaces with a single space.

「Practice3」

Charming maneuvering, Stellar Navigator!

Now, I want to see more of your Python prowess. Replace the missing parts (____), ensuring that they perform the right operations. We aim to transform all text to lowercase and eliminate email addresses, URLs, special characters, digits, and extra spaces.

import re

  

def clean_text(text):

text = text.____() # Convert text to lower case

text = re.sub(r'____', '', text) # Remove email addresses

text = re.sub(r'____', '', text) # Remove URLs

text = re.sub(r'____', ' ', text) # Remove punctuation and special characters

text = re.sub(r'____', '', text) # Remove digits

text = re.sub(r'____', ' ', text) # Remove extra spaces

  

return text.strip()

  

test_texts = ['Sample Text with EMAIL! mailsample@mail.com',

'URL example: https://www.codesignal.com',

'Special characters #formed.',

'Digits included: 1234!!',

'Extra spaces included.']

for text in test_texts:

print(f'Original: {text}')

print(f'Cleaned: {clean_text(text)}')

print('--')

Absolutely, let’s fill in the missing parts to ensure the clean_text function works as intended. The goal is to convert all text to lowercase and eliminate email addresses, URLs, special characters, digits, and extra spaces. Here’s the complete function:

import re

def clean_text(text):
    text = text.lower()  # Convert text to lower case
    text = re.sub(r'\S*@\S*\s?', '', text)  # Remove email addresses
    text = re.sub(r'http\S+', '', text)  # Remove URLs
    text = re.sub(r'\W', ' ', text)  # Remove punctuation and special characters
    text = re.sub(r'\d', '', text)  # Remove digits
    text = re.sub(r'\s+', ' ', text)  # Remove extra spaces

    return text.strip()

test_texts = ['Sample Text with EMAIL! mailsample@mail.com', 
              'URL example: https://www.codesignal.com', 
              'Special characters #formed.', 
              'Digits included: 1234!!',
              'Extra spaces   included.']

for text in test_texts:
    print(f'Original: {text}')
    print(f'Cleaned: {clean_text(text)}')
    print('--')

Explanation:

1. Convert text to lower case: text = text.lower()
2. Remove email addresses: text = re.sub(r'\S*@\S*\s?', '', text)
3. Remove URLs: text = re.sub(r'http\S+', '', text)
4. Remove punctuation and special characters: text = re.sub(r'\W', ' ', text)
5. Remove digits: text = re.sub(r'\d', '', text)
6. Normalize whitespace to single spaces: text = re.sub(r'\s+', ' ', text)

When you run this script, it will clean each text string according to the rules specified and print the original and cleaned text for each example. This ensures that the function performs the desired operations correctly.

output：

Original: Sample Text with EMAIL! mailsample@mail.com  
Cleaned: sample text with email  
--  
Original: URL example: https://www.codesignal.com  
Cleaned: url example  
--  
Original: Special characters #formed.  
Cleaned: special characters formed  
--  
Original: Digits included: 1234!!  
Cleaned: digits included  
--  
Original: Extra spaces   included.  
Cleaned: extra spaces included  
--

「Practice4」

Greetings, data pioneers! Let’s take our next step.

You have the people dataframe, wherein unique IDs are set as the index. However, it appears that we’re missing the details of the person with ID 503. Could you complete the code to display the specifics of the individual with ID 503, utilizing the lessons we have learned so far?

import pandas as pd

  

people = {

"Name": ["Charlie", "Oliver", "Sophie", "Lucas"],

"Age": [33, 27, 36, 38],

"City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]

}

  

df = pd.DataFrame(people)

df['ID'] = [501, 502, 503, 504]

df.set_index('ID', inplace=True)

  

# TODO: Print the details of the person with ID 503

Sure, let’s rephrase and expand your query to ensure we understand and provide a comprehensive solution.

Your goal is to retrieve and display the details of the individual with ID 503 from the people dataframe. The dataframe currently contains columns for “Name”, “Age”, and “City” with unique IDs as the index. We need to write the appropriate code to locate and print the details for the person with ID 503.

Here’s how we can do it:

1. Retrieve the specific row using the ID.
2. Print the details of the person with ID 503.

Here is the complete code to accomplish this:

import pandas as pd

# Provided data
people = {
    "Name": ["Charlie", "Oliver", "Sophie", "Lucas"],
    "Age": [33, 27, 36, 38],
    "City": ["Toronto", "Vancouver", "Ottawa", "Montreal"]
}

# Create DataFrame
df = pd.DataFrame(people)

# Add ID column and set it as the index
df['ID'] = [501, 502, 503, 504]
df.set_index('ID', inplace=True)

# Retrieve and print the details of the person with ID 503
person_503 = df.loc[503]
print(person_503)

When you run this code, it will output the details of the person with ID 503:

Name    Sophie
Age         36
City     Ottawa
Name: 503, dtype: object

This approach ensures that we accurately locate and display the details of the specific individual identified by the unique ID 503. 输出

Name    Sophie  
Age         36  
City    Ottawa  
Name: 503, dtype: object

Lesson 2: Implementing Bag-of-Words Representation

You’re doing an amazing job learning all about descriptive statistics and Python! I’m super excited for this lesson on central tendency. Let’s dive in and explore the mysteries of mean, median and mode! 🌟

Introduction to Descriptive Statistics and Python

Greetings, data enthusiast! Today, we are diving into descriptive statistics using Python. We’ll be exploring measures of centrality — mean, median, and mode — using Python libraries numpy and pandas.

Understanding Central Tendency

A central tendency finds a ‘typical’ value in a dataset. Our three components — the mean (average), median (mid-point), and mode (most frequently appearing) — each offer a unique perspective on centrality. The mean indicates average performance when decoding students’ scores, while the median represents the middle student’s performance, and the mode highlights the most common score.

Visualizing Central Tendency

This plot represents a given dataset’s mean or centered location, also considered the ‘average’. Imagine a seesaw balancing at its center - the mean of a dataset is where it balances out. It is a crucial statistical concept and visually helps identify where most of our data is centered around or leaning toward. 

Setting up the Dataset

Our dataset is a list of individuals’ ages: [23, 22, 22, 23, 24, 24, 23, 22, 21, 24, 23]. Remember, understanding your data upfront is key to conducting a meaningful analysis.

Computing Mean using Python

Calculating the mean involves adding all numbers together and then dividing by the count. Here’s how you compute it in Python:

import numpy as np

data = np.array([23, 22, 22, 23, 24, 24, 23, 22, 21, 24, 23])
mean = np.mean(data)  # calculates the mean
print("Mean: ", round(mean, 2))  # Mean:  22.82

Computing Median using Python

The median is the ‘middle’ value in an ordered dataset. This is how it is computed in Python:

import numpy as np

data = np.array([23, 22, 22, 23, 24, 24, 23, 22, 21, 24, 23])
median = np.median(data)  # calculates the median
print("Median: ", median)  # Median:  23.0

Computing Mode using Python

The mode represents the most frequently occurring number(s) in a dataset. To compute it, we use the mode function from the scipy library:

from scipy import stats

data = np.array([23, 22, 22, 23, 24, 24, 23, 22, 21, 24, 23])
mode_age = stats.mode(data)  # calculates the mode
print("Mode: ", mode_age.mode)  # Mode:  23

Note, that calculated mode_age is an object. To retrieve the actual value from it, we use the .mode attribute of this object. So, resulting line is mode_age.mode. NumPy doesn’t have a function for calculating mode, so we are using the SciPy module here. We will talk more about this module and its capabilities in the future lessons.

Handling Ties in Mode with scipy

Great job so far! Now let’s explore an interesting concept: how the mode function from scipy.stats handles ties or duplicate modes.

So, what’s a tie in mode? Imagine we have two or more different numbers appearing the same number of times in our dataset. For instance, consider this dataset: [20, 21, 21, 23, 23, 24]. Here, 21 and 23 both appear twice and are therefore modes.

Let’s calculate the mode using scipy.stats:

from scipy import stats
import numpy as np

data = np.array([20, 21, 21, 23, 23, 24])
mode = stats.mode(data)
print("Mode: ", mode.mode)  # Mode: 21

Although 21 and 23 are both modes, our calculation only returned 21. Why is that?

In cases of ties, scipy.stats.mode() returns the smallest value amongst the tied modes. So in this case, it picked 21 over 23 because 21 is the smaller value.

Choice of Measure of Central Tendency

Your choice of measure of central tendency depends on the nature of your data. For numerical data, the mean is susceptible to outliers, i.e., extreme values, making the median a preferable measure. The mode is undefined when no particular value repeats, or all values repeat with equal frequency. For categorical data, the mode is the only meaningful measure.

Wrapping Up

Kudos! You have mastered the measures of central tendency and have learned how to compute them using Python! Stay tuned for some hands-on exercises for deeper reinforcement. Onward!

「Practice 1」Analyzing Age Group Dataset with Measures of Centrality

Greetings, fellow explorer!

In our Age Group Analysis, we’ve calculated measures of centrality for a group of individuals’ ages. Let’s review!

Click the Run button to display the mean, median, and mode for the dataset.

import numpy as np

from scipy import stats

  

age_group = np.array([8, 9, 8, 10, 10, 9, 10, 10, 11, 8, 10])

mean_age_group = np.mean(age_group)

median_age_group = np.median(age_group)

mode_age_group = stats.mode(age_group)

  

print("Mean age: ", mean_age_group)

print("Median age: ", median_age_group)

print("Mode age: ", mode_age_group.mode)

解释 Thank you for pointing that out! Let’s correct the code by accessing the mode directly without indexing. Here is the updated version:

import numpy as np
from scipy import stats

# Define the dataset of ages
age_group = np.array([8, 9, 8, 10, 10, 9, 10, 10, 11, 8, 10])

# Calculate the mean of the ages
mean_age_group = np.mean(age_group)

# Calculate the median of the ages
median_age_group = np.median(age_group)

# Calculate the mode of the ages
mode_age_group = stats.mode(age_group)

# Print the results
print("Mean age: ", mean_age_group)
print("Median age: ", median_age_group)
print("Mode age: ", mode_age_group.mode)

Let’s run this corrected version to display the mean, median, and mode of the dataset.

知识点拆解

好的，让我们具体解释一下这个代码示例中的主要知识点。

数据集定义

age_group = np.array([8, 9, 8, 10, 10, 9, 10, 10, 11, 8, 10])